uttrflow / uttrflow/uttrflow-swift

In heavy noise the recogniser once turned a two-word command into a profanity, and returned nothing on the rerun

Open
#482 0 comments 0 reactions 0 assignees View on GitHub
area:dictation bug help wanted P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

In a noise-robustness run, one synthetic clip of a short two-word command, mixed with noise at 10 dB signal-to-noise ratio, was transcribed once as a near-homophone profanity, and as an empty transcript when the same clip was run again.

Decoding starts greedy (`temperature: 0`) and is retried up to five times, each 0.2 warmer, when Whisper's own thresholds reject a window (`Sources/UttrflowSpeech/VocabularyPrompt.swift:55-59`). Greedy decoding of the same audio is deterministic, so a different answer on a rerun points at the fallback path: sampled decoding at a higher temperature is the only source of randomness here.

## Why it matters

A profanity inserted into someone's email or chat, from audio that contained none, is among the worst errors a dictation tool can make, even if it is rare. That it is nondeterministic makes it hard to catch in a corpus.

## How to reproduce

- Take a short command-like phrase from the synthetic passages, mix it with noise at 10 dB SNR, and transcribe it many times (for example 50) with `uttrflow-dev transcribe `, counting distinct outputs.
- Log which temperature produced each output (WhisperKit reports fallbacks in its timings).

## Acceptance criteria

- The run is repeated enough times to say how often each outcome occurs, and whether the bad outputs come only from fallback temperatures.
- A decision is recorded, for example: reject a fallback result that introduces a word from a small profanity list that the greedy result did not contain; or prefer an empty result over a high-temperature one for very short clips. Any guard has tests with invented, non-offensive stand-ins where possible.
- The noisy clip is added to the evaluation corpus so a regression is caught.

`help wanted`: real recordings in noisy places would say whether this happens outside synthetic noise.

Contributor guide

Open the contributing guide

Research direction

Start with the fallback path referenced in Sources/UttrflowSpeech/VocabularyPrompt.swift:55-59, then run repeated transcriptions of a synthetic command at 10 dB SNR with uttrflow-dev transcribe and record outputs and fallback temperatures. Use invented stand-ins for guard tests, add the noisy clip to the evaluation corpus, and document which outcomes occur and the chosen handling policy.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, swift
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.