godotengine / godotengine/godot
AudioEffectCapture with surround sound audio artifacts
- Dominant language
- C++
- Stars
- 117k
- Forks
- 26.8k
- PR merge metrics
- PR metrics pending
Description
### Tested versions
4.2.2-stable and 4.2.1-stable, but likely all others since the relevant code hasn't changed in years.
### System information
Godot v4.2.2.stable - Windows 10.0.19045 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 2080 Ti (NVIDIA; 31.0.15.4633) - Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz (8 Threads) - WASAPI using IAudioClient3
### Issue description
AudioEffectCapture with a Surround speaker mode (i.e. more than one stereo channel) causes audio artifacts.
The `AudioServer`'s `_mix_step()` function processes bus effects by iterating over each active channel. For each channel, it calls the `process()` function of each effect, passing the channel's buffer as an argument.
The `AudioEffectCapture::process` function processes the source buffer by copying it to the destination buffer and then, if there is available space in the capture ring buffer, copies the source buffer into it.
What this means for surround audio setups is that after the first 512 (i.e. the hard coded `buffer_size`) stereo channel frames are copied into the capture buffer, the extra 3.1, 5.1, 7.1 channels will get copied in afterwards. In my 7.1 setup, the higher channels are muted during the `_mix_step_for_channel()` (`channel_vol` is 0.0), so my capture buffer ends up with 512 good audio frames, then 512*3 zeroed audio frames, which results in horrible popping/crackling artifacts in the audio. Even if the surround channels were not muted, the issue would still occur because the capture buffer would still be filled with consecutive frames from different channels, causing artifacts.
This happens regardless of the `AudioStream` type (sound files, mic, generator/procedural).
A potential solution to this issue could be to modify the `AudioEffectInstance::process` function to accept the channel index as an additional parameter. This would allow each effect to handle the source frames differently based on the channel index. Capture would probably only want the first channel, or maybe a weighted average of all channels. Other effects could just ignore the channel index to continue behaving the same as they do now.
A drawback of this solution is that it would require effects to be aware of the meaning of each channel index, which could couple the concept of channel indices across different effects (i.e. channel 0 is regular stereo left/right, channel 1 is surround 3.1 LFE and center, channel 2 is 5.1 rear left/right, etc.). Right now all channels seem to be independent, treated the same by most of the audio code. There may be another, better way.
### Steps to reproduce
The most important aspect needed to test/reproduce the issue is a surround sound audio setup (i.e. more than just a standard stereo channel pair). The Dummy audio driver seems like it can simulate surround speaker modes, but setting speaker mode isn't accessible from GDScript or the editor, as far as I know, so I couldn't set it up in the MRP below.
Assuming a surround sound speaker mode, configure an audio bus with a Capture effect on it. Add an AudioStreamPlayer with a sound file or some other stream (the MRP uses a Generator populated with a sine wave), set the bus to Capture.
Attach a script that reads frames from the capture effects buffer 512 frames at a time. Each batch of 512 frames should come from one channel in the audio bus. The first batch would be the main stereo sound (i.e. the sound we probably want), while the next few will be either empty (all zero) or possibly other surround channels, sequentially packed into the capture buffer.
The resulting frames can be inspected for the above pattern, or passed on to a Generator for output to hear the artifacts directly.
### Minimal reproduction project (MRP)
[CaptureAudioIssue.zip](https://github.com/godotengine/godot/files/15103185/CaptureAudioIssue.zip)
Contributor guide
Research direction
Start at AudioServer::_mix_step(), _mix_step_for_channel(), and AudioEffectCapture::process() to trace how surround channels enter the capture ring buffer. Reproduce the issue with the linked MRP and a surround speaker mode, then determine a capture behavior that preserves coherent frames without disrupting other effects. Done means captured audio no longer contains sequential channel blocks that cause artifacts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- audio-video-rtc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100