uttrflow / uttrflow/uttrflow-swift
In heavy noise the recogniser once turned a two-word command into a profanity, and returned nothing on the rerun
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## What happens
In a noise-robustness run, one synthetic clip of a short two-word command, mixed with noise at 10 dB signal-to-noise ratio, was transcribed once as a near-homophone profanity, and as an empty transcript when the same clip was run again.
Decoding starts greedy (`temperature: 0`) and is retried up to five times, each 0.2 warmer, when Whisper's own thresholds reject a window (`Sources/UttrflowSpeech/VocabularyPrompt.swift:55-59`). Greedy decoding of the same audio is deterministic, so a different answer on a rerun points at the fallback path: sampled decoding at a higher temperature is the only source of randomness here.
## Why it matters
A profanity inserted into someone's email or chat, from audio that contained none, is among the worst errors a dictation tool can make, even if it is rare. That it is nondeterministic makes it hard to catch in a corpus.
## How to reproduce
- Take a short command-like phrase from the synthetic passages, mix it with noise at 10 dB SNR, and transcribe it many times (for example 50) with `uttrflow-dev transcribe `, counting distinct outputs.
- Log which temperature produced each output (WhisperKit reports fallbacks in its timings).
## Acceptance criteria
- The run is repeated enough times to say how often each outcome occurs, and whether the bad outputs come only from fallback temperatures.
- A decision is recorded, for example: reject a fallback result that introduces a word from a small profanity list that the greedy result did not contain; or prefer an empty result over a high-temperature one for very short clips. Any guard has tests with invented, non-offensive stand-ins where possible.
- The noisy clip is added to the evaluation corpus so a regression is caught.
`help wanted`: real recordings in noisy places would say whether this happens outside synthetic noise.
Contributor guide
Research direction
Start with the fallback path referenced in Sources/UttrflowSpeech/VocabularyPrompt.swift:55-59, then run repeated transcriptions of a synthetic command at 10 dB SNR with uttrflow-dev transcribe and record outputs and fallback temperatures. Use invented stand-ins for guard tests, add the noisy clip to the evaluation corpus, and document which outcomes occur and the chosen handling policy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, swift
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100