uttrflow / uttrflow/uttrflow-swift

dictation_bench.py splits every Devanagari word at its vowel signs, so Hindi word error rates are counted over letter fragments and a missing or extra matra scores as correct

Open Beginner friendly
#705 0 comments 0 reactions 0 assignees View on GitHub
area:dictation bug good first issue P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

`normalise()` in `Scripts/dictation_bench.py:265-276` removes punctuation with

```python
t = re.sub(r"[^\w\s]", " ", t.replace("'", "").replace("’", ""))
```

Python's `\w` doesn't match combining marks. Devanagari vowel signs (ा ि ी ु ू े ै ो ौ), anusvara (ं), chandrabindu (ँ), virama (्) and nukta (़) are categories Mn/Mc, so each one becomes a space and every word is cut into pieces:

```
>>> normalise("मैं नहीं आऊँगा, पचास लाख।")
['म', 'नह', 'आऊ', 'ग', 'पच', 'स', 'ल', 'ख']
>>> errors(["कल शाम को मैं घर जल्दी पहुँच गया"], "कल शाम को मै घर जल्दी पहुंच गया")
(0, 11) # 8 words, 2 of them misspelt, scored as 11 fragments with 0 errors
>>> errors(["मैं नहीं आऊँगा"], "मैं नहीं आऊँगा बिल्कुल")
(4, 4) # one extra word scored as 100%
```

## Measured

The same bench run (`origin/main` 26d7bc1, the 12 committed Hindi and Hinglish passages read by `say -v Lekha`) scored two ways:

| | today's `normalise` | punctuation removed by Unicode category instead |
|---|---|---|
| Hindi, 6 passages | 9.0% | 13.9% |
| Hinglish, 6 passages | 33.9% | 24.2% |
| `hi-everyday` | 2.3% | 6.7% |
| `hinglish-people` | 43.6% | 22.6% |

So Hindi accuracy is reported about a third better than it is, and Hinglish worse. Any comparison between the two, or between a Devanagari and a romanised answer, is meaningless.

(The Swift scorer in `Sources/UttrflowEval/TextNormaliser.swift` splits on letters and marks correctly; this is the Python bench only.)

## Why it matters

The bench is what a speech or clean-up change is judged by (`Docs/performance.md`). For Hindi it can't tell a dropped anusvara from a correct word, and it inflates the word count it divides by.

## Acceptance criteria

- `normalise()` treats a character as punctuation by its Unicode category (`unicodedata.category(ch)[0] in "PS"`) rather than `[^\w\s]`, so marks stay inside their word.
- The three examples above give `['मैं', 'नहीं', 'आऊँगा', 'पचास', 'लाख']`, `(2, 8)` and `(1, 3)`.
- English scoring is unchanged: rerun `score` on an existing English run and get identical numbers.
- `Docs/performance.md` notes that Hindi figures from before the fix aren't comparable.

## Where to start

- `Scripts/dictation_bench.py:265-276` (`normalise`) and `:288` (`errors`)
- There are no Python tests. Check with `python3 -c "import sys; sys.path.insert(0, 'Scripts'); import dictation_bench as d; print(d.normalise('मैं नहीं आऊँगा'))"`, and with `python3 Scripts/dictation_bench.py score ` on a run if you have one.
- Still run `make verify` before pushing (export `DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer` first).
- Read [CONTRIBUTING.md](https://github.com/uttrflow/uttrflow-swift/blob/main/CONTRIBUTING.md) first, and say on this issue that you are taking it.

**Size:** XS, about 1 hour.

Contributor guide

Open the contributing guide

Research direction

Read CONTRIBUTING.md, then inspect normalise() at Scripts/dictation_bench.py:265-276 and errors() at :288. Run the supplied python3 -c Hindi example and compare the three acceptance examples, then use an existing run with score if available. Update Docs/performance.md and confirm English results are unchanged; finish with make verify using the stated DEVELOPER_DIR.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.