uttrflow / uttrflow/uttrflow-swift
dictation_bench.py splits every Devanagari word at its vowel signs, so Hindi word error rates are counted over letter fragments and a missing or extra matra scores as correct
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## What happens
`normalise()` in `Scripts/dictation_bench.py:265-276` removes punctuation with
```python
t = re.sub(r"[^\w\s]", " ", t.replace("'", "").replace("’", ""))
```
Python's `\w` doesn't match combining marks. Devanagari vowel signs (ा ि ी ु ू े ै ो ौ), anusvara (ं), chandrabindu (ँ), virama (्) and nukta (़) are categories Mn/Mc, so each one becomes a space and every word is cut into pieces:
```
>>> normalise("मैं नहीं आऊँगा, पचास लाख।")
['म', 'नह', 'आऊ', 'ग', 'पच', 'स', 'ल', 'ख']
>>> errors(["कल शाम को मैं घर जल्दी पहुँच गया"], "कल शाम को मै घर जल्दी पहुंच गया")
(0, 11) # 8 words, 2 of them misspelt, scored as 11 fragments with 0 errors
>>> errors(["मैं नहीं आऊँगा"], "मैं नहीं आऊँगा बिल्कुल")
(4, 4) # one extra word scored as 100%
```
## Measured
The same bench run (`origin/main` 26d7bc1, the 12 committed Hindi and Hinglish passages read by `say -v Lekha`) scored two ways:
| | today's `normalise` | punctuation removed by Unicode category instead |
|---|---|---|
| Hindi, 6 passages | 9.0% | 13.9% |
| Hinglish, 6 passages | 33.9% | 24.2% |
| `hi-everyday` | 2.3% | 6.7% |
| `hinglish-people` | 43.6% | 22.6% |
So Hindi accuracy is reported about a third better than it is, and Hinglish worse. Any comparison between the two, or between a Devanagari and a romanised answer, is meaningless.
(The Swift scorer in `Sources/UttrflowEval/TextNormaliser.swift` splits on letters and marks correctly; this is the Python bench only.)
## Why it matters
The bench is what a speech or clean-up change is judged by (`Docs/performance.md`). For Hindi it can't tell a dropped anusvara from a correct word, and it inflates the word count it divides by.
## Acceptance criteria
- `normalise()` treats a character as punctuation by its Unicode category (`unicodedata.category(ch)[0] in "PS"`) rather than `[^\w\s]`, so marks stay inside their word.
- The three examples above give `['मैं', 'नहीं', 'आऊँगा', 'पचास', 'लाख']`, `(2, 8)` and `(1, 3)`.
- English scoring is unchanged: rerun `score` on an existing English run and get identical numbers.
- `Docs/performance.md` notes that Hindi figures from before the fix aren't comparable.
## Where to start
- `Scripts/dictation_bench.py:265-276` (`normalise`) and `:288` (`errors`)
- There are no Python tests. Check with `python3 -c "import sys; sys.path.insert(0, 'Scripts'); import dictation_bench as d; print(d.normalise('मैं नहीं आऊँगा'))"`, and with `python3 Scripts/dictation_bench.py score ` on a run if you have one.
- Still run `make verify` before pushing (export `DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer` first).
- Read [CONTRIBUTING.md](https://github.com/uttrflow/uttrflow-swift/blob/main/CONTRIBUTING.md) first, and say on this issue that you are taking it.
**Size:** XS, about 1 hour.
Contributor guide
Research direction
Read CONTRIBUTING.md, then inspect normalise() at Scripts/dictation_bench.py:265-276 and errors() at :288. Run the supplied python3 -c Hindi example and compare the three acceptance examples, then use an existing run with score if available. Update Docs/performance.md and confirm English results are unchanged; finish with make verify using the stated DEVELOPER_DIR.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100