Audio (3D?)

[mfep]

OK, I managed to put in some effort this week, and am happy to share some results.

• The audio backend has been abstracted away, so we can now support multiple backends, and switch between them at runtime.
• The original audio system has been refactored into the so called SDL backend.
• And a new backend has been added named OpenAL backend.
• The menu has been modified has been added to switch between the two, with the option of enable binaural rendering in the OpenAL backend's case.

Notably, I didn't change how the rest of the game systems interact with the audio system, nor any of the audio files. Still, I have the impression that the result is a major improvement over the current (SDL) backend. The number of concurrently playing sources is greatly improved, and the very noticable clicking when stealing voices in the original audio system is gone. Moreover, the very ugly hard clipping of loud signals is also fixed using OpenAL. Also, the other effects I mentioned above should also be present. I had to tune down the Doppler effect a bit: using the real-world speed of sound sounded a bit too much. This can be refined later, or even be controlled by other in-game systems.

To illustrate the results, here is a short gameplay recording in which I switch multiple times between the two backends to compare the two: https://youtu.be/Pp1bdDPJvio

For now, this works on Linux only. I will have to make sure that Windows is also supported, and that OpenAL Soft is properly integrated to the build pipeline.

[mfep]

For the record, a year later this work has been merged to master. With that, the game uses the OpenAL Soft audio backend by default. However, the merged PR was only a minimal changeset to create the new backend, without too much modifications in other areas of the codebase. Below, I attempt to collect what else could be done to improve the game's audio from both the player and the developer's perspective. I hope that some sort of "audio roadmap" might fall out of this. Feel free to add your thoughts!

• As mentioned elsewhere, currently the audio listener is tied to the player ship's position and orientation. Most of the time, this works OK, but when the camera is not aligned with the ship or we're in the external camera view, this is obviously not good. Implementing this would be quite straightforward by defining an audio listener, that can be set to other transforms than the player's ship.
• One caveat of OpenAL is that it works with mono sources for spatialized audio (there are downmix/upmix extensions in OpenAL Soft, but that is not relevant to the point). This is inherent in the programming model of OpenAL, and the fact that it must work with any output configuration, e.g. stereo, 5.1, etc. However, the Sound API in Pioneer offers stereo panning, in the OpenAL sound backend, this is simulated by moving the (mono) sound source in a circle around the listener. There are two places where this is used in the game currently: weapons and most notably the thruster audio. It would improve the audio realism if these were converted to "physical" audio sources at the location where the actual thrusters/lasers are, and the audio engine would take care of how those sound in the cockpit. In effect, external (non-player) ships would gain thruster audio trivially. A side benefit would be a great deal of simplification possible in the audio backend's implementation.
• Bus mixing: the current bus setup is rudimentary: we have the master bus, and the music and the SFX bus is routed to it. Providing flexible bus setups would be beneficial to soundscape design, and also defining common operations on buses, such as pause state, is sometimes required by the game logic (see e.g. PR #6333). While OpenAL Soft does not have buses, it is not difficult to implement on top of OpenAL sources.
• More tools for sound design. Allowing access to OpenAL effects would open quite some possibilities. Sample randomization, e.g. by selecting from a pre-defined set is also quite feasible and could go a long way.
• A general refactoring of the current Sound interface to something akin to something like the state-of-the-art game engines offer, an AudioPlayer component per say, offering a single voice with possible looping. The current Sound::Event is similar, but that functionality is not exposed to Lua. In effect, sounds invoked from Lua are always one-shots that limits what can be achieved artistically. Arguably, a single interface for all audio, including SFX, ambience and music - and the same API exposed to scripts - would be beneficial.
• Sample-accurate level automation. This is somewhat of a nice-to-have. The current logic updates every sound source at every frame, and that applies to the level fades too. In effect, especially if the framerate is unstable, this can result in missing fades. A good example is the first frame when launching the menu. Here, the menu music should come in a faded manner, but instead it cuts in immediately. This is because the first frame takes longer than the actual fade duration. Implementing this would require modifying the audio stream fed into the source, instead of updating the Gain parameter of the source. Quite a lot of work for maybe not much benefit.

[impaktor]

The list is sorted based on priority / relevance?

First few suggestions sounds good to me, the others I'm not fully understanding, some seem like it might perhaps drain more resources for little added benefit, or perhaps audio is cheap? As you can tell, I know nothing about audio code.

If you fear an implementation might be difficult to review, then having clean commit history would help (i.e. not 10 features in one commit).