uttrflow / uttrflow/uttrflow-swift
Hinglish accuracy needs real speakers: synthetic Hindi speech loses English loanwords to Devanagari in 5 of 6 passages and mishears लाख in 3 of 3
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## What happens
The only Hindi audio this repository can measure with is the system's synthetic `Lekha` voice. On it, the recogniser does two things a Hindi or Hinglish speaker would notice immediately. Nobody can yet say whether real speech does the same, because no native-speaker recordings exist (`Docs/measuring-accuracy.md`: "The audio: Missing").
## Measured
`origin/main` at 26d7bc1, Release `uttrflow-dev bench`, fast mode, the 12 committed `TranscriptionCorpus` Hindi and Hinglish passages read by `say -v Lekha`. Word error rate was scored per word, with the Devanagari splitting in `dictation_bench.py` corrected (#705). Everything was tidied by the rules, as #445 describes, so these are the recogniser's words.
| | passages | word error rate | notes |
|---|---|---|---|
| Hindi | 6 | 13.9% (6.1–25.8%) | `hi-numbers` 25.8%: घंता, घेज, आत, मदवा |
| Hinglish | 6 | 24.2% (6.7–32.3%) | loanwords written in Devanagari |
**Loanwords.** The Hinglish passages' comment says the Devanagari reference "keep[s] borrowed words in Latin script, as the recogniser does" (`Sources/UttrflowEval/TranscriptionCorpus.swift:184`). On this audio it did that in 1 of 6 passages (`hinglish-terms`). In the other 5, the `mustKeep` terms came back transliterated or misheard: deploy → डिपला, issue → इशू, staging → स्टिजिंग, refund → रिफांद, PR → पी आर, debug → दीबार, doctor → डॉप्टर, pending → पैंडिंग. A romanising clean-up can't turn डिपला back into "deploy".
**Lakh.** In 3 short clips of amounts, लाख came back as लाक or लाग every time ("50 लाक रुपए", "2 करोड़ 50 लाग", "देर लाक रोपे" for डेढ़ लाख रुपये).
The synthetic voice gives English words Hindi phonology and has no natural code-switching prosody. So these may be properties of the voice rather than the recogniser. That is the question.
## What would help
Recordings from native Hindi and Hinglish speakers of the committed passages. The tooling already exists and works offline:
1. Build: `swift build -c release --product uttrflow-eval` (see CONTRIBUTING.md).
2. `uttrflow-eval record` walks through the passages in `TranscriptionCorpus.swift`. Read the 6 Hinglish passages, and the 6 Hindi ones if you can, in your normal voice and room.
3. `uttrflow-eval transcribe` scores them. Please report the per-passage word error rate, which `mustKeep` words were lost, and in which script each loanword came back. Also say your speaking background (for example "Hindi first language, Delhi"), your microphone, and the macOS version.
4. Optionally, also say "पचास लाख रुपये" and "दो करोड़ पचास लाख" a few times each.
**Please don't attach the audio files or anything personal to this issue.** Numbers and the words as recognised are enough.
## Acceptance criteria
- At least two speakers' results for the Hinglish passages are reported here.
- If the loanword and lakh failures reproduce on real speech, a follow-up issue is filed for the recogniser side (for example vocabulary prompting with the passage's loanwords, which `VocabularyPrompt` already supports). If they don't reproduce, the corpus comment at `TranscriptionCorpus.swift:184` is kept and this note about the synthetic voice goes into `Docs/measuring-accuracy.md`.
Contributor guide
Research direction
Build the release evaluator with `swift build -c release --product uttrflow-eval`, then use `uttrflow-eval record` and `transcribe` for the passages in `Sources/UttrflowEval/TranscriptionCorpus.swift`. Report per-passage word error rates, lost `mustKeep` terms and their scripts for at least two speakers, along with speaking background, microphone and macOS version; record the appropriate follow-up or documentation result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, swift
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100