uttrflow / uttrflow/uttrflow-swift

Hinglish accuracy needs real speakers: synthetic Hindi speech loses English loanwords to Devanagari in 5 of 6 passages and mishears लाख in 3 of 3

Open
#708 0 comments 0 reactions 0 assignees View on GitHub
area:dictation enhancement help wanted P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

The only Hindi audio this repository can measure with is the system's synthetic `Lekha` voice. On it, the recogniser does two things a Hindi or Hinglish speaker would notice immediately. Nobody can yet say whether real speech does the same, because no native-speaker recordings exist (`Docs/measuring-accuracy.md`: "The audio: Missing").

## Measured

`origin/main` at 26d7bc1, Release `uttrflow-dev bench`, fast mode, the 12 committed `TranscriptionCorpus` Hindi and Hinglish passages read by `say -v Lekha`. Word error rate was scored per word, with the Devanagari splitting in `dictation_bench.py` corrected (#705). Everything was tidied by the rules, as #445 describes, so these are the recogniser's words.

| | passages | word error rate | notes |
|---|---|---|---|
| Hindi | 6 | 13.9% (6.1–25.8%) | `hi-numbers` 25.8%: घंता, घेज, आत, मदवा |
| Hinglish | 6 | 24.2% (6.7–32.3%) | loanwords written in Devanagari |

**Loanwords.** The Hinglish passages' comment says the Devanagari reference "keep[s] borrowed words in Latin script, as the recogniser does" (`Sources/UttrflowEval/TranscriptionCorpus.swift:184`). On this audio it did that in 1 of 6 passages (`hinglish-terms`). In the other 5, the `mustKeep` terms came back transliterated or misheard: deploy → डिपला, issue → इशू, staging → स्टिजिंग, refund → रिफांद, PR → पी आर, debug → दीबार, doctor → डॉप्टर, pending → पैंडिंग. A romanising clean-up can't turn डिपला back into "deploy".

**Lakh.** In 3 short clips of amounts, लाख came back as लाक or लाग every time ("50 लाक रुपए", "2 करोड़ 50 लाग", "देर लाक रोपे" for डेढ़ लाख रुपये).

The synthetic voice gives English words Hindi phonology and has no natural code-switching prosody. So these may be properties of the voice rather than the recogniser. That is the question.

## What would help

Recordings from native Hindi and Hinglish speakers of the committed passages. The tooling already exists and works offline:

1. Build: `swift build -c release --product uttrflow-eval` (see CONTRIBUTING.md).
2. `uttrflow-eval record` walks through the passages in `TranscriptionCorpus.swift`. Read the 6 Hinglish passages, and the 6 Hindi ones if you can, in your normal voice and room.
3. `uttrflow-eval transcribe` scores them. Please report the per-passage word error rate, which `mustKeep` words were lost, and in which script each loanword came back. Also say your speaking background (for example "Hindi first language, Delhi"), your microphone, and the macOS version.
4. Optionally, also say "पचास लाख रुपये" and "दो करोड़ पचास लाख" a few times each.

**Please don't attach the audio files or anything personal to this issue.** Numbers and the words as recognised are enough.

## Acceptance criteria

- At least two speakers' results for the Hinglish passages are reported here.
- If the loanword and lakh failures reproduce on real speech, a follow-up issue is filed for the recogniser side (for example vocabulary prompting with the passage's loanwords, which `VocabularyPrompt` already supports). If they don't reproduce, the corpus comment at `TranscriptionCorpus.swift:184` is kept and this note about the synthetic voice goes into `Docs/measuring-accuracy.md`.

Contributor guide

Open the contributing guide

Research direction

Build the release evaluator with `swift build -c release --product uttrflow-eval`, then use `uttrflow-eval record` and `transcribe` for the passages in `Sources/UttrflowEval/TranscriptionCorpus.swift`. Report per-passage word error rates, lost `mustKeep` terms and their scripts for at least two speakers, along with speaking background, microphone and macOS version; record the appropriate follow-up or documentation result.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, swift
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.