uttrflow / uttrflow/uttrflow-swift
Apple's model answers a short Hindi dictation with its own worked example, or with an English translation, and the guard accepts both
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## What happens
When Apple's model accepts a short Devanagari dictation (#445 covers the longer ones it declines), what it returns is sometimes not the dictation. `MeaningPreservationGuard` compares no Devanagari tokens (#196), so the answer is accepted and inserted.
## Measured
`origin/main` at 26d7bc1, Release `uttrflow-dev bench`, shipping router, fast mode, macOS 26.5.1. Clips were made with `say -v Lekha`. Each was run 3 times, and all 3 answers were identical.
| heard (recogniser) | inserted, `by=foundationModels` | what went wrong |
|---|---|---|
| मतलब मैं कल आएगा, हाँ, अच्छा तो फिर मिलते हैं. | Main aaj ke standup mein deployment ke baare mein baat karunga. | **Not the dictation.** This is the worked example at `Sources/UttrflowAI/PromptContract.swift:53-54`, word for word. |
| मीटिंग चार बजे है, नहीं नहीं, पांच बजे है. | Meeting is at four o'clock, no no, five o'clock. | Translated into English. The prompt forbids this (`PromptContract.swift:18-20`). |
| मेरी फ्लाइट 15 अगस्त को सुबह सारले 9 बजे है। | My flight 15 August morning Sarale 9 AM. | Translated, and "AM" added. |
| वो क्या है ना, यानि मुझे थोड़ा ताइम चाहिए. | Woh kya hai na, yaani mujhe thoda time chahiye. | Fine: romanised as intended. |
| आज बहुत बारिश हो रही है। (and 3 other short sentences) | Aaj bahut baarish ho rahi hai. | Fine. |
Of 15 short Hindi clips, 11 were answered by the model: 8 romanised as intended, 2 translated into English, and 1 replaced by the example. Every one was accepted.
The replacement shares no word with the input and is exactly one of the prompt's `WorkedExample`s. The translation keeps the digits, so the guard's digit check passes.
## Why it matters
A Hindi speaker says "I'll come tomorrow, then see you" and a sentence about a standup and a deployment is typed into their chat. That is invented content, the failure `Docs/cleanup.md` puts above every other. Translation into English is the other thing the contract says must never happen.
The Style page tells Hindi users they "get punctuation and spacing, not rewriting" (`Sources/UttrflowUX/StylePagePresentation.swift:117-120`), so nobody would think to check.
## How to reproduce
```
say -v Lekha -o f.wav --file-format=WAVE --data-format=LEI16@16000 "मतलब मैं कल आऊँगा, हाँ, अच्छा तो फिर मिलते हैं।"
printf 'f\t%s\t\tfast\tshipping\n' "$PWD/f.wav" > jobs.tsv
uttrflow-dev bench jobs.tsv
```
Or, without audio: `uttrflow-dev clean -e foundationModels "मतलब मैं कल आएगा, हाँ, अच्छा तो फिर मिलते हैं."`
## Acceptance criteria
- An answer equal to, or mostly made of, a worked example's `cleaned` text that the dictation did not contain is rejected. This check doesn't depend on script, so it works before #196's tokeniser work.
- A Devanagari dictation whose answer is English prose (Latin letters, and none of the romanised forms of the Hindi words) is rejected, or recorded as undecided and handed to the rules, rather than accepted.
- Both cases join the clean-up corpus (`Sources/UttrflowEval/EvaluationCorpus.swift`, multilingual section) with `mustNotAdd` for the example's words and for "o'clock".
- The Style page callout matches what Hindi dictation actually gets.
Contributor guide
Assessment
This issue has not been assessed yet.