uttrflow / uttrflow/uttrflow-swift
A dictation that opens in English holds every later piece to English, so a Hindi sentence after a pause is translated into invented English or dropped
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## What happens
The first piece of a dictation that reports a language fixes it for every later piece (`Sources/UttrflowPipeline/DictationPipeline.swift:572`, `:594`):
```swift
let language = dictationLanguage
...
if dictationLanguage == nil { dictationLanguage = transcription.detectedLanguage?.code }
```
That was the fix for #177, and it is right for one language. But Hinglish speakers switch language between sentences. When the first sentence is English and the next is Hindi, the Hindi piece is decoded under the English token. Whisper then does not transcribe it: it translates it, loosely, or it returns nothing.
## Measured
`origin/main` at 26d7bc1, Release `uttrflow-dev bench`, rules cleaner, macOS 26.5.1. The clip (12.6 s) was made with `say`: an English sentence in the `Rishi` voice spoken twice, a 1.5 s pause, then one Hindi sentence in the `Lekha` voice:
> कल मैं ऑफिस नहीं आऊँगा, घर से काम करूँगा, और शाम तक रिपोर्ट भेज दूँगा।
It was cut into two pieces (6.6 s English, 6.0 s Hindi). The second piece always ran with `language: en`.
| run | mode | what the Hindi piece became |
|---|---|---|
| 1 | rt | `""`: the sentence is gone, and nothing says so |
| 2 | rt | "I will not have office, I will work on the day and the morning I will report to you. I will not be a office, I will not have work on the day and I will not have report on the day." |
| 3 | rt | "I will not have office, I will work out and I will be a report to the evening." |
| 4 | fast | "I will not get office, I will work on the show, and the report will be sent to me." |
So in 4 of 4 runs the Hindi sentence did not arrive as said. In 3 it was replaced by English the speaker never said, including a repeated clause; in 1 it was dropped. The rules cleaner inserted it unchanged.
Reversed (Hindi first, then English), the English piece is decoded under `hi` and comes back in English, but one of the two identical English sentences is missing in 3 of 3 runs.
The recogniser is at fault for the words, but the pipeline is what forces the wrong language onto it.
## Why it matters
Switching language between sentences is how Hinglish is spoken ("Okay, quick update. कल मैं…"). Here the speaker's sentence is silently rewritten into English with a different meaning, or lost. Nothing downstream can recover it, and nothing in the result shows that it happened.
## How to reproduce
```
say -v Rishi -o a.wav --file-format=WAVE --data-format=LEI16@16000 "Okay team, quick update on the release today."
say -v Lekha -o b.wav --file-format=WAVE --data-format=LEI16@16000 "कल मैं ऑफिस नहीं आऊँगा, घर से काम करूँगा, और शाम तक रिपोर्ट भेज दूँगा।"
# join a + 0.5 s silence + a + 1.5 s silence + b into one 16 kHz mono WAV
printf 'switch\t%s\t\trt\trules\n' "$PWD/joined.wav" > jobs.tsv
uttrflow-dev bench jobs.tsv # the second "asr" event has language "en"
```
## Acceptance criteria
- A later piece whose own detection disagrees with the held language, with enough confidence, is decoded in its own language. Alternatively, the held language is a prior rather than a hard hint: for example, detection is still run and the held language only breaks close calls. The #177 case (a short or noisy piece flipping language) must stay fixed.
- A piece that decodes to empty text while its audio held speech is not silently dropped.
- A test in `Tests/UttrflowPipelineTests/DictationPipelineLanguageTests.swift` covers an English piece followed by a Hindi piece.
- A code-switch clip is added to `Scripts/dictation_bench.py` so this is measured again (#707).
Contributor guide
Assessment
This issue has not been assessed yet.