ml-explore / ml-explore/mlx-examples
whisper: align no_speech_threshold fallback condition with openai/whisper
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.2k
- PR merge metrics
- No merged PRs in 30d
Description
Summary
The MLX Whisper decode_with_fallback logic appears to differ from upstream openai/whisper in how no_speech_threshold suppresses fallback decoding.
In upstream OpenAI Whisper, high no_speech_prob only disables fallback when the average log probability is also below logprob_threshold:
if (
no_speech_threshold is not None
and decode_result.no_speech_prob > no_speech_threshold
and logprob_threshold is not None
and decode_result.avg_logprob < logprob_threshold
):
needs_fallback = False # silence
In mlx_whisper.transcribe, the condition is broader:
if (
no_speech_threshold is not None
and decode_result.no_speech_prob > no_speech_threshold
):
needs_fallback = False # silence
Why this matters
This means MLX Whisper can accept a decode as “silence” solely because no_speech_prob is high, even if fallback was triggered for another reason such as high compression ratio / repetitive output.
The upstream behavior is narrower: it only suppresses fallback when the segment looks like silence according to both high no-speech probability and low average log probability. That may affect hallucination/repetition behavior.
References
OpenAI Whisper source:
https://github.com/openai/whisper/blob/main/whisper/transcribe.py
Relevant upstream condition is in decode_with_fallback.
MLX Whisper source:
whisper/transcribe.py, inside decode_with_fallback.
Suggested change
Match upstream OpenAI Whisper by adding the logprob_threshold and avg_logprob checks:
if (
no_speech_threshold is not None
and decode_result.no_speech_prob > no_speech_threshold
and logprob_threshold is not None
and decode_result.avg_logprob < logprob_threshold
):
needs_fallback = False # silence
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in whisper/transcribe.py at decode_with_fallback and compare its no_speech_threshold logic with the referenced OpenAI Whisper condition. Update the fallback decision so silence requires both high no-speech probability and low average log probability, then verify that the resulting behavior still handles compression-ratio or repetition-triggered fallback correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100