decentraland / decentraland/unity-explorer

Perceived 2-3 seconds of delay in nearby voice chat

Open
#10,087 1 comment 0 reactions 0 assignees View on GitHub
1-high bug
Dominant language
C#
Stars
23
Forks
17
Avg merge
2d 14h
Merged PRs (30d)
94

Description

## Summary

Users experience a perceived 2–3 second delay in nearby voice chat. Code analysis of the voice pipeline in `unity-explorer` and the pinned LiveKit fork (`decentraland/client-sdk-unity` @ `a4bd750`) explains why.

## Pipeline analysis

The mic-to-speaker path accumulates latency at each hop:

| Hop | Component | Estimated latency |
|---|---|---|
| 1. Capture | OS mic → Rust `cpal` callback → `bounded(1)` channel → C# thread (`RustAudioClient.Capture`) | ~10–40ms |
| 2. Framing | 200ms `NativeAudioBuffer` chunked into 10ms frames + APM (echo cancel/AGC/noise suppression) in `MicrophoneRtcAudioSource.OnAudioRead` | ~10–20ms |
| 3. Send | Opus encode → network → LiveKit SFU → network | ~60–150ms |
| 4. Receive | WebRTC adaptive jitter buffer (NetEq, inside the Rust FFI) | elastic |
| 5. Playback | FFI `FrameReceived` → 200ms ring buffer (`NativeAudioBuffer`, ~341ms capacity after power-of-2 rounding) → `OnAudioFilterRead` → DSP output | ~250–380ms |

**Designed steady-state latency is ~0.4–0.7s.** A constant 2–3s means one of the elastic components is growing — almost certainly the **NetEq jitter buffer on the receiver**, which expands when frame arrival is irregular.

## Root causes of irregularity

### 1. Frame drops at capture (`RustAudioClient.cs`)
The Rust mic callback uses `try_send` into a `bounded(1)` channel. If the C# consumer thread is preempted for even one callback interval, audio frames are **silently dropped**. Gaps in the outgoing stream cause NetEq on the receiver side to grow its buffer to compensate, which never shrinks back during a session.

```rust
// rust-audio/src/lib.rs — capture callback
let _ = channel.0.try_send(data); // drop on full — no retry, no backpressure
```

### 2. Main-loop contention stalls the FFI event pump
Voice chat is used in exactly the situations where the client is busiest (crowded scenes). Frame hitches stall the FFI event dispatch thread, so remote audio frames arrive in bursts. The playback ring buffer (`NativeAudioBuffer`, 200ms target, ~341ms actual after power-of-2 rounding) **overwrites its oldest samples** when a burst exceeds its capacity — inserting more discontinuities that NetEq then pads against.

### 3. All-or-nothing playback reads (`NativeRingBuffer.cs`)
`NativeRingBuffer.TryDequeue` only succeeds if the ring buffer has a full DSP-buffer chunk worth of samples. When it falls short by even one sample it returns silence and waits for the next callback. This causes the buffer to run near-full (i.e. at the top of its latency range) rather than running lean.

```csharp
// NativeAudioBuffer.cs
Span read = buffer.TryDequeue((int)lengthToRead, out bool success);
if (success) { /* play */ }
// else: output silence, accumulate more buffering on next callback
```

### 4. Additive serial buffering
The ~200ms send-side chunker + ~200–341ms receive-side ring buffer + jitter buffer all stack additively. None of them share slack with each other.

## How to confirm

The SDK already ships the right instrumentation:
- `AudioStream.WavTeeControl` can dump the **network-side received stream** to disk.
- `MicrophoneRtcAudioSource.WavTeeControl` can dump the **raw and resampled mic** stream.

Comparing timestamps between the mic dump on the speaker's machine and the network dump on the listener's machine will isolate exactly which hop holds the 2s. LiveKit's `jitterBufferDelay` / `jitterBufferEmittedCount` stats from the FFI would confirm NetEq growth directly.

## Suggested fixes (in impact order)

1. **Fix the capture channel** (`RustAudioClient` / `rust-audio/src/lib.rs`): increase `bounded(1)` to `bounded(4–8)` or use a blocking send with a short timeout so mic frames are not silently dropped under momentary thread preemption.

2. **Allow partial playback reads** (`NativeAudioBuffer.cs` / `NativeRingBuffer.cs`): return whatever samples are available and fade them in (a short ramp of ~5ms is enough to prevent clicks) rather than outputting silence when the buffer is slightly short. This allows the ring buffer to run lean instead of near-full.

3. **Instrument jitter-buffer depth** (`AudioStream.cs`): expose `LastFrameReceivedAt` delta or jitter-buffer stats to Sentry/Grafana so the latency is measurable per session rather than anecdotal.

## Relevant files

- `RustAudio/rust-audio/src/lib.rs` — capture channel bounded(1)
- `RustAudio/Wrap/RustAudioClient.cs` — C# capture loop
- `Runtime/Scripts/Audio/NativeAudioBuffer.cs` — send-side and receive-side ring buffer
- `Runtime/Scripts/Types/NativeRingBuffer.cs` — TryDequeue all-or-nothing
- `Runtime/Scripts/RtcSources/Audio/MicrophoneRtcAudioSource.cs` — APM + framing
- `Runtime/Scripts/Rooms/Streaming/Audio/AudioStream.cs` — receive-side ring buffer + FFI event handler
- `Runtime/Scripts/Rooms/Streaming/Audio/LivekitAudioSource.cs` — OnAudioFilterRead playback

(Files above are in `decentraland/client-sdk-unity` @ `a4bd750e2262b9eeeee75dc83858017bbd3679ef`, the version pinned in `unity-explorer`.)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.