lablup / lablup/mlxcel

feat(nemotron_voicechat): full-duplex speech model (FastConformer + Nemotron-H + EAR-TTS + codec) with a /v1/realtime session

Open
#1,372 0 comments 0 reactions 0 assignees View on GitHub
arch:hybrid area:cli area:docs area:inference area:models modelsize:small modeltype:audio priority:medium status:ready type:enhancement
Dominant language
Rust
Stars
467
Forks
54
Avg merge
4h 25m
Merged PRs (30d)
310

Description

## Summary

NemotronLabs VoiceChat (`mlx-community/NemotronLabs-VoiceChat-11B-{bf16,8bit,4bit}`, `model_type: "nemotron_voicechat"`, `architectures: ["NemotronVoiceChatForConditionalGeneration"]`) is a full-duplex speech-to-speech model. It consumes 16 kHz audio in 80 ms frames and, on every frame, emits an assistant text token, a function-channel token, a user-transcript update, and 80 ms of 22.05 kHz assistant speech, all on one shared timeline (silence and overlap are part of the timeline; there is no VAD). The model is four networks in one checkpoint: a cache-aware FastConformer speech encoder with an RNNT transcript branch, a 56-layer Nemotron-H hybrid LLM with two output heads, an EAR-TTS speech decoder (Gemma-3-style sliding-window backbone, character-aware text conditioning, mixture-of-Gaussians RVQ refinement), and a 31-codebook neural audio codec. mlxcel has nothing for this family; it does have Nemotron-H, Gemma-3 blocks, an audio worker thread, WAV I/O, a log-mel frontend, and an LSTM and STFT/iSTFT pair inside Kokoro that the sub-issues reuse.

## Motivation

- mlxcel serves STT (Whisper) and TTS (Kokoro) as separate request/response endpoints (`src/server/whisper_stt.rs`, `src/server/kokoro_tts.rs`, `src/server/audio_worker.rs`). A duplex model is a different product surface: a stateful, bidirectional session where audio flows both ways continuously. Landing it establishes the streaming-session abstraction (per-connection model state, frame clock, aligned event stream) that any later speech-to-speech family will reuse.
- The checkpoint's 4-bit conversion is 9.2 GB and the per-frame work is bounded by caches, so a single Apple Silicon machine can hold a conversation in real time; this is the first mlxcel model where wall-clock throughput per frame is a correctness requirement, not just a benchmark number.
- The LLM is Nemotron-H (`hybrid_override_pattern` with `M`, `-`, `*` blocks), which `src/models/nemotron_h.rs` already runs with the fused Mamba2 kernels; reusing it keeps the largest component off the new-code path.

## Sub-issues

1. #1374: load the checkpoint and run the four networks offline on a WAV file (speech encoder + RNNT, Nemotron-H with fused text/audio/function input and two heads, EAR-TTS warmup and per-step code generation, codec encode/decode). Delivers `mlxcel generate -m --audio input.wav` printing the transcript and assistant text and writing `response.wav`.
2. #1378 (depends on 1): the stateful online session that turns arbitrary PCM chunks into 1280-sample frames and advances every network with persistent caches (streaming log-mel, FastConformer attention/conv caches, Nemotron-H sequence state, TTS KV caches, codec causal-conv and iSTFT caches), emitting aligned events with frame indices; includes the system-prompt prefill, flush/pad, cancel, and a per-frame stage profiler.
3. #1376 (depends on 2): the `/v1/realtime` WebSocket endpoint on `mlxcel-server` (session.created / session.update / input_audio_buffer.append / commit / cancel / ping, base64 PCM16 both ways, one active session, a dedicated MLX worker thread), plus a CLI microphone/speaker example.

## Delivery order

1 -> 2 -> 3. Sub-issue 1 is validated offline against the checkpoint's published behavior; 2 is validated by first-frame parity with 1; 3 is validated by a WebSocket client driving a WAV file through the endpoint and comparing the concatenated audio events with the offline output.

## Definition of done

- `mlxcel-server -m models/NemotronLabs-VoiceChat-11B-4bit` starts, a client connects to `ws://host/v1/realtime`, sends `session.update` with a system prompt, streams base64 PCM16 at 16 kHz in arbitrary chunk sizes, receives `response.text.delta`, `conversation.item.input_audio_transcription.delta`, `response.function.delta`, and `response.audio.delta` (22.05 kHz PCM16) events with monotonically increasing `frame_index`, and ends with `response.done` after `input_audio_buffer.commit`.
- A second concurrent connection is refused with `server_busy` while the first is active and accepted after it closes.
- The offline CLI path and the streaming path produce identical tokens and audio codes for the first frame of the same input (parity test), and the offline path transcribes and answers a spoken question in the checkpoint's built-in `Aria` voice.
- Per-frame latency is measured and reported by the profiler (`perception`, `rnnt`, `language`, `tts`, `codec`, `total` in ms, p50/p95), and the README for the family states the measured real-time factor on the validation machine for the 4-bit and 8-bit checkpoints.
- `docs/supported-models.md` lists the family under speech models; `mlxcel list` reports it.

## Out of scope

- Voice cloning or any voice other than the checkpoint's `Aria` prompt latents.
- Acoustic echo cancellation in the microphone example.
- Batch size above 1 and more than one concurrent realtime session.
- Converting the original NeMo `.nemo` checkpoint; mlxcel loads the converted safetensors layout only.

Contributor guide

Open the contributing guide

Research direction

Start with sub-issue #1374 and read src/models/nemotron_h.rs plus the existing audio paths in src/server/whisper_stt.rs, src/server/kokoro_tts.rs, and src/server/audio_worker.rs. First validate the offline CLI flow against the checkpoint, then follow sub-issues #1378 and #1376; done requires the stated offline/streaming parity, realtime events, concurrency behavior, profiling, and documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
audio-video-rtc, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.