uttrflow / uttrflow/uttrflow-swift

The "Languages you speak" setting is never read by recognition, and short Hindi replies are decoded as English words: 4 of 10 synthetic clips

Open
#699 1 comment 0 reactions 1 assignee Claimed by @naveen-bhatt View on GitHub
area:dictation bug needs-info P1
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

**The setting.** Settings › Languages and the Style page show "Languages you speak", with English and Hindi rows (`Sources/UttrflowUX/SettingsChoices.swift:131-134`). Unticking the last one is refused with "Uttrflow needs at least one language to listen for." (`Sources/UttrflowUX/SettingsPresenter.swift:237-262`, `SettingsEditor.swift:236-252`).

Recognition never reads it:

- `UserProfile.preferredLanguages` is read only by those two screens and by the fallback in `Sources/UttrflowCore/Models/Transformation.swift:34`.
- Every dictation starts with no language hint (`Sources/UttrflowPipeline/DictationPipeline.swift:325`). The first piece detects freely between `LanguageCode.transcribed = [.english, .hindi]` (`Sources/UttrflowCore/Models/LanguageCode.swift:33`, `Sources/UttrflowSpeech/WhisperKitBackend.swift:53`).

So "Hindi only" still detects English, and "English only" still detects Hindi. `Docs/speech-engines.md` ("The constraint is the product's languages, not the profile's") explains why the profile isn't used: it defaults to English for everyone. But the screen still says it decides what Uttrflow listens for.

**The effect.** Language detection on a one- or two-second reply is unreliable, and these are the dictations where a Hindi speaker would most want to say "Hindi".

## Measured

`origin/main` at 26d7bc1, Release `uttrflow-dev bench`, fast mode, rules cleaner. Ten short Hindi replies made with `say -v Lekha`:

| said | inserted | language |
|---|---|---|
| हाँ ठीक है। | Happy K. | en |
| हाँ जी। | Hagi. | en |
| मैं आ रहा हूँ। | May I have a hug. | en |
| धन्यवाद। | Thanhyevad! | en |
| नहीं। | नहीं. | hi |
| ठीक है। | ठीक है. | hi |
| अच्छा ठीक है। | अच्छा ठीक है. | hi |
| कोई बात नहीं। | कोई बात नहीं. | hi |
| बस पाँच मिनट। | बस पांच मिनत. | hi |
| चलो ठीक है, कल मिलते हैं। | चलो ठीक है, कल मिलते हैं. | hi |

4 of 10 came back as English words, one of them ("May I have a hug.") with a different and awkward meaning. Word error rate over the ten is 46.4%. The recogniser's language detection is at fault; clean-up passed the text through.

The voice is synthetic, so the rate for real speakers is unknown. The four failures are exactly the replies that sound like English syllables.

## Why it matters

"Haan theek hai", "haan ji" and "main aa raha hoon" are among the most frequent things a Hindi speaker dictates into a chat. Getting "Happy K." or "May I have a hug." in a message is worse than getting nothing. The one control that looks like it would prevent it does nothing.

## Decision needed

Which of these should the setting do?

1. Hold detection to the ticked languages. Only a profile with a single language is affected, and the default could become both languages rather than English.
2. Use the ticked languages as a prior: for example, a clip under about 3 s from a profile that ticks Hindi is decoded as Hindi unless English wins clearly.
3. Keep today's behaviour and change the copy so it doesn't say the setting decides what is listened for.

## Acceptance criteria

- The chosen behaviour is implemented, and the Languages copy matches it.
- A test in `Tests/UttrflowPipelineTests/DictationPipelineLanguageTests.swift` shows the profile reaches `TranscriptionOptions` (or pins that it deliberately doesn't).
- Short Hindi replies are added to the bench corpus (#707), and the 4-of-10 figure is re-measured.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.