uttrflow / uttrflow/uttrflow-swift

Apple's model answers a short Hindi dictation with its own worked example, or with an English translation, and the guard accepts both

Open
#700 1 comment 0 reactions 1 assignee Claimed by @naveen-bhatt View on GitHub
area:dictation bug P1
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

When Apple's model accepts a short Devanagari dictation (#445 covers the longer ones it declines), what it returns is sometimes not the dictation. `MeaningPreservationGuard` compares no Devanagari tokens (#196), so the answer is accepted and inserted.

## Measured

`origin/main` at 26d7bc1, Release `uttrflow-dev bench`, shipping router, fast mode, macOS 26.5.1. Clips were made with `say -v Lekha`. Each was run 3 times, and all 3 answers were identical.

| heard (recogniser) | inserted, `by=foundationModels` | what went wrong |
|---|---|---|
| मतलब मैं कल आएगा, हाँ, अच्छा तो फिर मिलते हैं. | Main aaj ke standup mein deployment ke baare mein baat karunga. | **Not the dictation.** This is the worked example at `Sources/UttrflowAI/PromptContract.swift:53-54`, word for word. |
| मीटिंग चार बजे है, नहीं नहीं, पांच बजे है. | Meeting is at four o'clock, no no, five o'clock. | Translated into English. The prompt forbids this (`PromptContract.swift:18-20`). |
| मेरी फ्लाइट 15 अगस्त को सुबह सारले 9 बजे है। | My flight 15 August morning Sarale 9 AM. | Translated, and "AM" added. |
| वो क्या है ना, यानि मुझे थोड़ा ताइम चाहिए. | Woh kya hai na, yaani mujhe thoda time chahiye. | Fine: romanised as intended. |
| आज बहुत बारिश हो रही है। (and 3 other short sentences) | Aaj bahut baarish ho rahi hai. | Fine. |

Of 15 short Hindi clips, 11 were answered by the model: 8 romanised as intended, 2 translated into English, and 1 replaced by the example. Every one was accepted.

The replacement shares no word with the input and is exactly one of the prompt's `WorkedExample`s. The translation keeps the digits, so the guard's digit check passes.

## Why it matters

A Hindi speaker says "I'll come tomorrow, then see you" and a sentence about a standup and a deployment is typed into their chat. That is invented content, the failure `Docs/cleanup.md` puts above every other. Translation into English is the other thing the contract says must never happen.

The Style page tells Hindi users they "get punctuation and spacing, not rewriting" (`Sources/UttrflowUX/StylePagePresentation.swift:117-120`), so nobody would think to check.

## How to reproduce

```
say -v Lekha -o f.wav --file-format=WAVE --data-format=LEI16@16000 "मतलब मैं कल आऊँगा, हाँ, अच्छा तो फिर मिलते हैं।"
printf 'f\t%s\t\tfast\tshipping\n' "$PWD/f.wav" > jobs.tsv
uttrflow-dev bench jobs.tsv
```

Or, without audio: `uttrflow-dev clean -e foundationModels "मतलब मैं कल आएगा, हाँ, अच्छा तो फिर मिलते हैं."`

## Acceptance criteria

- An answer equal to, or mostly made of, a worked example's `cleaned` text that the dictation did not contain is rejected. This check doesn't depend on script, so it works before #196's tokeniser work.
- A Devanagari dictation whose answer is English prose (Latin letters, and none of the romanised forms of the Hindi words) is rejected, or recorded as undecided and handed to the rules, rather than accepted.
- Both cases join the clean-up corpus (`Sources/UttrflowEval/EvaluationCorpus.swift`, multilingual section) with `mustNotAdd` for the example's words and for "o'clock".
- The Style page callout matches what Hindi dictation actually gets.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.