Audio (3D?)

[FluffyFreak]

Just starting the discussion for whether or not we should/could/would switch too an actual audio library rather than the homegrown system currently in use.

There various discussions around the place: https://arstechnica.com/civis/viewtopic ... 0&t=122852

And various open source libraries:
http://www.attr-x.net/yse/
https://github.com/MarcusTomlinson/Ambisonix

Along with stuff that's free for non-commercial use:
https://www.ambiera.com/irrklang/features.html

Currently our audio is a bit lacking in terms of what it can do and is strictly 2D.

I usually play with the sound turned off almost entirely :/

[impaktor]

Well, if someone does the work of implementing it, and if it would then be used in the game for something, and license is compatible, I'm fine with it.

But there's no 3D sound in space, I assume? Or when would 3D sound be used?

[bszlrd]

In Star Wars, at least in the old EU canon I think it was explained that the sound is actually from the ship computer providing an audio from what's happening around you, to provide that familiar sensory layer for the pilot for orientation beyond sight.
Which kinda makes, especially for a Star Wars thing.

[sturnclaw]

3d sound is something that we should be implementing, if for nothing other than atmospheric flight. I'll throw OpenAL Soft in the hat as well, it's LGPL-licensed and a fairly standard piece of software.

https://github.com/kcat/openal-soft

[bszlrd]

http://spacesimcentral.com/forums/topic ... dio-plans/
Might be useful.

[mfep]

Hi all,

I'd like to express interest on picking this topic up. Looking at the Pioneer source code, it appears to be a manageable task to add a new audio back-end with leaving invocations of the sound system untouched, so this would be a fairly localized update. Also, it's a good idea to leave the option to switch between the original audio system and this proposed new one at runtime.

Consensus here appears to be that a 3rd party 3D audio library is desirable, and I also share this opinion. Just to clarify what would be gained from this, if we choose the right library for the task:

• Currently, only a sound source's relative horizontal position to the listener affects the playback, and the vertical position has no influence. While in this is very light to compute, a better immersion can be offered by the usage of HRTF filters, which colours the sound based on the source's direction (horizontal and vertical). Not all libraries mentioned here offer HRTF support, and I would rule those out, because those would not really be an improvement.
• Most of the libraries feature support for velocity-based effects, such as the Doppler effect. This would change the pitch of sound sources based on relative velocity to the listener. Some libraries also consider the time the sound takes to propagate to the listener, which can also result in quite cool effects (one very memorable for me is the "seismic charge" scene from Attack of the Clones). Of course all these have to be tuned carefully to achieve the desired artistic effect.
• Usually, the libraries offer spatial output rendering, meaning that our players would be able to enjoy Pioneer not only in stereo, but also in multichannel audio. Don't only think of a 5.1 home theatre, some gamer headphones also support multi-channel audio.

Also I briefly looked at the candidate libraries mentioned here or elsewhere. Only candidates with OSI approved licences were considered. While some proprietary tools are free to use and might offer better features or tooling, I'd say using such would potentially put the project in jeopardy in the long term. For completeness, let's start with the ones I ruled out for one or the other reason:

• https://github.com/cross-platform/ambisonix Doesn't appear to be a serious or well maintained project.
• https://solhsa.com/soloud/ Very friendly API and documentation, but unfortunately no HRTF.
• https://github.com/NotAPenguin0/audeo A bit hobby-category, also no HRTF.
• https://github.com/pirogronian/audio3d Also a hobby-category wrapper over OpenAL Soft, I'm not convinced.

And here come the serious contenders:

• https://github.com/kcat/openal-soft As mentioned previously, this is a fairly standard piece. It has an impressive feature set and supports a number of backends. Documentation is somewhat scattered around, especially that some functionality is supported via extensions. The API appears to be somewhat unintuitive to me (caveat : I'm also not familiar with OpenGL it is supposed to be based on).
• https://github.com/ValveSoftware/steam-audio As the name suggests, this is Valve's project, but the license is permissive. This appears to be focusing on live generation of local ambience using raytracing (most probably Pioneer does not want this), but also it has HRTF. Documentation is nice.

So the next step for me is probably play around with the serious contenders, trying to figure which one is more suitable. I'm very open to suggestions about my approach or about libraries not on my radar. Also, it would be very nice to get a confirmation from the project maintainers that such work is welcome. Thank you for reading this far!

[zonkmachine]

Hi all,

Hi!

But there's no 3D sound in space, I assume? Or when would 3D sound be used?

When a ship is hit for instance. The sound would travel via hull/atmosphere/(shield?), depending on the weapon/hit, to the cabin. In a real world scenario you would experience this in 3D. The way we experience up/down (3D) in hearing is through something called 'spectral cues'. I learned about 3D sound in this video from youtuber Smarter Every Day, https://www.youtube.com/watch?v=Oai7HUqncAA, in which an adult engineer stuffs play dough into the ears of his son in the name of science.

[sturnclaw]

mfep, welcome! I definitely appreciate the desire to improve Pioneer's sound system, as it's probably the single most primitive subsystem in the entire game (with particle effects being a very close second contender).

> Not all libraries mentioned here offer HRTF support, and I would rule those out, because those would not really be an improvement.

On the contrary, any of the below would be quite the improvement. We currently have no support for doppler effects or any proper filters, and a very contrarian API when it comes to managing sound playback.

> While some proprietary tools are free to use and might offer better features or tooling[...]

Keep in mind that any library chosen here would have to be compatible with Pioneer's GPLv3 licensing, so you're looking at a fully-permissible license (MIT, BSD, Apache, ZLib) or a GPL variant.

Of the contenders you ruled out, all but soloud are ones I would also rule out (but we'll get to that soon). Audeo in particular is a very tiny hobby-grade wrapper around SDL_Mixer.

Of the libraries you marked as serious contenders:

- OpenAL Soft is a seemingly viable implementation, but is rushing pell-mell after new C++ versions (having tied themselves to C++20 as a minimum version within the last few months). If we were to integrate that codebase, it would need to be pinned to a specific older version and accept that we won't get any updates. I'm also concerned about the compile-time impact of integrating the project, as it does not appear to be a particularly lightweight library. The complete reliance on vcpkg is also a concern as to how much build system troubleshooting we would need to do.

We could potentially look at linking against OpenAL Soft but not distributing it ourselves, though the Windows build would need to include a binary as a redistributable.

- Steam Audio is a *very* heavyweight library, both in terms of codebase size as well as feature set. We will almost certainly never need the raytraced audio occlusion that is the "killer feature" of its implementation. That being said, it does look like it can be built without dragging in all of the raytracing dependencies, so it is still an option (of last resort).

- The third option is to integrate soloud and either upstream or maintain our own fork which adds an HRTF filter. The main difficulty there seems to be sourcing the dataset for the filter and supporting the file format used to encode the HRTF data. I'm not sure this is necessarily worth the effort - as long we can dynamically link against OpenAL Soft and *not* have to do in-tree builds.

[mfep]

When a ship is hit for instance. The sound would travel via hull/atmosphere/(shield?), depending on the weapon/hit, to the cabin. In a real world scenario you would experience this in 3D. The way we experience up/down (3D) in hearing is through something called 'spectral cues'. I learned about 3D sound in this video from youtuber Smarter Every Day, https://www.youtube.com/watch?v=Oai7HUqncAA, in which an adult engineer stuffs play dough into the ears of his son in the name of science.

Yep, these spectral cues are partly what is captured by the HRTF (head related transfer function). This method is only really effective with headphones though. To keep the topic narrow, I didn't go into detail regarding the lore. An idea from the top of my head: there could be a ship-installable equipment module that would provide this "simulated" audio in space. Also, it could be cool for immersion to deduce settings of the audio engine (e.g. speed of sound) from the game environment, such as air pressure. But first we need the tech for this.

sturnclaw, thank you for the answer and confirmation!

Keep in mind that any library chosen here would have to be compatible with Pioneer's GPLv3 licensing, so you're looking at a fully-permissible license (MIT, BSD, Apache, ZLib) or a GPL variant.

OK, noted.

From your post, I read that you find OpenAL Soft an acceptable dependency, with some caveats regarding integration. Here are my two cents on the matter:

The README of OpenAL Soft is somewhat misleading, since it only lists source install and vcpkg (which is also a wrapped source install) as ways to integrate the library. This may be true for the HEAD, but also they publish release artifacts for Windows (we can directly use that to link against and redistribute). For Linux distributions, more-or-less recent dev versions of the library are also found in the system repositories (https://repology.org/project/openal-soft/versions). Note, that the basic interface of OpenAL is quite narrow, and more esoteric features are implemented as extensions. The availability of the extensions can be queried at runtime, therefore I wouldn't be particularly afraid of coding against the latest release, and letting Pioneer to link against an older version on some distributions.

If in any case we'd need to do an in-tree build: here are some numbers to go by. On my PC, using Ninja (multi-threaded build) Pioneer builds in 57s, while OpenAL Soft builds in 11s (without pruning off the build of tests, examples and unneeded backends). A significant addition, but does not a change in the order of magnitude.

I agree with your point about Stream Audio.

I'm not a big fan of the idea of adding HRTF support to SoLoud, mostly due to this being a bit too much work in addition to updating Pioneer's audio backend. I don't see HRTF data itself a concern, if nothing else, we can use the MIT dataset from 1994. An alternative would be to just go with SoLoud, and defer HRTF. However, at the current moment I see that more value (player audible features) could be gained from using OpenAL Soft, even if it is somewhat more difficult to integrate.

[sturnclaw]

An 11s increase in build time for a 3rd-party library is an acceptable (though far from ideal) tradeoff. I was expecting significantly worse given their file count. Consider this an official blessing to integrate OpenAL as a backend.