block / block/buzz

Huddle: receive-side Opus in-band FEC for lossy relay paths

Open
#4,177 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Context

Investigated grainy (not laggy) huddle audio over an ordinary residential-ISP path (see internal discussion in #general on the web3services Buzz instance, 2026-08-01). Root cause: the relay's per-peer audio fan-out channel and the client's NetEq jitter buffer both drop/conceal frames under real jitter, and there's currently no recovery mechanism for a genuinely lost frame — only concealment (PLC/expand).

## What's already scoped in-repo

`desktop/src-tauri/src/huddle/jitter.rs:73-76`:

> Receive-side FEC ("decode this frame's redundant copy on a known-lost prior frame") would require either a trait change upstream or NetEq calling decode twice with different `fec` values — out of scope for the initial 10-person fix. Tracked as a follow-up alongside encoder-side `set_inband_fec`.

## Why this needs upstream work, not just a buzz-side change

Confirmed against `neteq` 0.9.0 (latest, we pin 0.8) via docs.rs: `neteq::codec::AudioDecoder::decode(&mut self, encoded: &[u8]) -> Result>` still takes no `fec` parameter and there's no fec-aware decode entry point. Opus in-band FEC only works if the *decoder* is explicitly asked to decode the redundant copy embedded in the *next* packet after a detected loss — the `opus` crate's `Decoder::decode_float` already supports this (it takes a `fec: bool` third argument; buzz's `OpusFrameDecoder::decode` in jitter.rs hardcodes `false`), but nothing in the `neteq` crate's loss-concealment path can currently reach it.

So a real fix needs one of:
1. A fork/patch of the `neteq` crate adding a fec-aware decode path (and using it when NetEq's own loss detection fires), or
2. Upstreaming that capability to `neteq` itself (crates.io owner: darioalessandro).

Either way it also needs encoder-side `opus::Encoder::set_inband_fec(true)` in `desktop/src-tauri/src/huddle/relay_api.rs` — enabling that alone does nothing without the receive-side change, so they must land together.

## Stopgap already shipped

Relay-side per-peer audio channel capacity was widened 8→16 frames (160ms→320ms) in `crates/buzz-relay/src/audio/room.rs` to reduce hard `try_send` drops under jitter close to the old ceiling. This does not recover lost audio content — it just reduces how often frames are dropped outright. See PR (link once open).

Contributor guide

Open the contributing guide

Research direction

Start with desktop/src-tauri/src/huddle/jitter.rs:73-76 and inspect neteq 0.8's AudioDecoder API and loss-concealment path. Then review desktop/src-tauri/src/huddle/relay_api.rs for encoder configuration and determine whether an upstream neteq change or a fork is needed. Done means receive-side Opus FEC and encoder-side in-band FEC work together; the relay channel widening is already shipped.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
audio-video-rtc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.