ml-explore / ml-explore/mlx-examples

whisper: align no_speech_threshold fallback condition with openai/whisper

Open Beginner friendly
#1,427 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
9k
Forks
1.2k
PR merge metrics
No merged PRs in 30d

Description

Summary

The MLX Whisper decode_with_fallback logic appears to differ from upstream openai/whisper in how no_speech_threshold suppresses fallback decoding.

In upstream OpenAI Whisper, high no_speech_prob only disables fallback when the average log probability is also below logprob_threshold:

if (
    no_speech_threshold is not None
    and decode_result.no_speech_prob > no_speech_threshold
    and logprob_threshold is not None
    and decode_result.avg_logprob < logprob_threshold
):
    needs_fallback = False  # silence

In mlx_whisper.transcribe, the condition is broader:

if (
    no_speech_threshold is not None
    and decode_result.no_speech_prob > no_speech_threshold
):
    needs_fallback = False  # silence

Why this matters

This means MLX Whisper can accept a decode as “silence” solely because no_speech_prob is high, even if fallback was triggered for another reason such as high compression ratio / repetitive output.

The upstream behavior is narrower: it only suppresses fallback when the segment looks like silence according to both high no-speech probability and low average log probability. That may affect hallucination/repetition behavior.

References

OpenAI Whisper source:
https://github.com/openai/whisper/blob/main/whisper/transcribe.py

Relevant upstream condition is in decode_with_fallback.

MLX Whisper source:
whisper/transcribe.py, inside decode_with_fallback.

Suggested change

Match upstream OpenAI Whisper by adding the logprob_threshold and avg_logprob checks:

if (
    no_speech_threshold is not None
    and decode_result.no_speech_prob > no_speech_threshold
    and logprob_threshold is not None
    and decode_result.avg_logprob < logprob_threshold
):
    needs_fallback = False  # silence

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in whisper/transcribe.py at decode_with_fallback and compare its no_speech_threshold logic with the referenced OpenAI Whisper condition. Update the fallback decision so silence requires both high no-speech probability and low average log probability, then verify that the resulting behavior still handles compression-ratio or repetition-triggered fallback correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.