github / github/copilot-cli

Voice mode: all bundled ASR models fail silently — MultiModalProcessor routing bug for nemotron_speech (RNNT) in Foundry Local Core

Đang mở
#4,024 13 bình luận 0 reaction 0 người được giao Xem trên GitHub
area:models
Ngôn ngữ chính
Shell
Star
11.2k
Fork
1.9k
Merge trung bình
14 giờ 16 phút
Pull request đã merge (30 ngày)
6

Mô tả

### Describe the bug

`/voice` records audio successfully (level meter reacts, mic capture confirmed via raw PulseAudio capture) but **every transcription comes back empty**, for **all three** models offered in the `/voice` model picker:
- `nemotron-3.5-asr-streaming-0.6b`
- `nemotron-speech-streaming-en-0.6b`
- `nemotron-speech-streaming-es-0.6b`

All three share the same `nemotron_speech` (RNNT) architecture, so switching models does not help — there is no working model in the current picker.

### Root cause (traced to source)

The Foundry Local Core native audio transcription path throws on every call:

```
Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: MultiModalProcessor cannot be created. nemotron_speech is not a registered multi-modal model type.
at Microsoft.ML.OnnxRuntimeGenAI.Result.VerifySuccess(IntPtr)
at Microsoft.ML.OnnxRuntimeGenAI.MultiModalProcessor..ctor(Model)
at Microsoft.AI.Foundry.Local.AudioClient.Transcribe(String, String, Nullable`1)
```

This is **not** a stale/outdated `onnxruntime-genai` engine issue. I downloaded and compared the public `onnxruntime-genai` v0.14.0 release (published 2026-05-29, after the "Multilingual Streaming Nemotron ASR" PR merged 2026-05-22) against the bundled runtime (labeled `0.14.1` in `deps_versions.json`, but not a publicly-tagged release). Both contain full, identical `NemotronSpeechModel`/`NemotronSpeechState` support (confirmed via symbol inspection of `libonnxruntime-genai.so`).

Cross-checking against `microsoft/onnxruntime-genai`'s `src/models/model.cpp` (https://github.com/microsoft/onnxruntime-genai/blob/main/src/models/model.cpp) confirms this is architecturally expected:

- `nemotron_speech` is an **RNNT-type** model (`ModelType::IsRNNT`) and is correctly routed to `NemotronSpeechModel` when the model is **loaded** (`CreateModel()` in `model.cpp`) — this is why `Model loaded successfully: nemotron-3.5-asr-streaming-0.6b-generic-cpu:3` appears in the Foundry log and the model shows as loaded.
- However, `MultiModalProcessor`'s constructor uses a **separate, hardcoded `processor_factory_` registry** that only contains vision+text chat model types (e.g. `phi4mm`, `gemma4`, etc.) — RNNT/TDT/ALM (Whisper) audio model types were never meant to go through `MultiModalProcessor` at all.

So the bug is that **`Microsoft.AI.Foundry.Local.Core`'s `AudioClient.Transcribe`/streaming session path unconditionally constructs a `MultiModalProcessor` for audio transcription**, instead of dispatching RNNT/TDT/ALM model types (`nemotron_speech`, Parakeet TDT, Whisper) through their own dedicated processing path (the same one used successfully at model-load time). Since `Microsoft.AI.Foundry.Local.Core.so` is a closed-source component, this can't be worked around or patched client-side — no `onnxruntime-genai` version swap fixes it, since the engine itself already works correctly and simply refuses (correctly, by its own design) to construct a `MultiModalProcessor` for a non-multimodal-chat model type.

### Affected version

GitHub Copilot CLI 1.0.69-0 (also reproduced on 1.0.66-1 — same bundled runtime pins: `onnxruntime-genai` 0.14.1, `foundry-local-core` 1.2.3, so the bug is not CLI-version-specific).

### Steps to reproduce the behavior

1. On Linux x64 (reproduced under WSL2/WSLg, but likely affects any Linux x64 install), run `/voice`, enable it, select any of the three offered models.
2. Press the voice-record shortcut, speak clearly for several seconds (confirmed via raw `parec` capture that real speech is captured at healthy signal levels, and the in-app level meter visibly reacts).
3. Stop recording.

### Expected behavior

Spoken audio is transcribed to text and inserted into the input box.

### Actual behavior

No error is shown to the user; the recording UI behaves normally, but the transcript is always empty. `~/.github-copilot-cli/logs/foundry.core*.log` shows:
```
[ERR] Error executing audio_transcribe: Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: MultiModalProcessor cannot be created. nemotron_speech is not a registered multi-modal model type.
...
Audio stream stop: session , final flush text: '', full transcript: ''
```

### Additional context

- Confirmed mic capture and PulseAudio pipeline are fully healthy (not a WSLg/audio-hardware issue) via direct raw `parec -d RDPSource` capture showing strong, correctly-timed speech signal.
- Confirmed via a standalone script using the bundled `foundry-local-sdk` package directly (bypassing the Copilot CLI JS entirely) that `audioClient.transcribe()` throws the identical `MultiModalProcessor`/`nemotron_speech` error against a real captured WAV file, ruling out any Copilot-CLI-specific bug in the JS layer.
- Suggested fix: in Foundry Local Core's audio transcription path, dispatch by `ModelType::IsRNNT/IsTDT/IsALM` (mirroring `CreateModel()`'s dispatch in `onnxruntime-genai`'s `model.cpp`) instead of unconditionally constructing a `MultiModalProcessor`.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Start with the Foundry Local Core AudioClient.Transcribe and streaming session path described in the report, then compare its routing with microsoft/onnxruntime-genai/src/models/model.cpp and the CreateModel dispatch. Done means the affected RNNT, TDT, and ALM models use their dedicated processing paths and /voice returns transcription instead of an empty result.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
linux
Lĩnh vực
ai, audio-video-rtc, backend
Loại issue
Lỗi
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
25/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.