uttrflow / uttrflow/uttrflow-swift
dictation_bench.py scores email addresses, identifiers, symbols and punctuation as correct when they are not
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## What happens
`Scripts/dictation_bench.py` reports one metric, word error rate after `normalise()`, which lower-cases, drops all punctuation, rewrites `x@y` as "x at y", splits camelCase and `snake_case`, turns `/` into "slash" and a dot between letters into "dot". That is right for "were the words heard", but it makes several categories report 0% or near it while every clip in them is wrong in the inserted text:
| Category (bench on main) | Reported final WER | What was inserted |
|---|---|---|
| `emails` | 2.2% in `Docs/performance.md` | "support at example.com" instead of "support@example.com" in all 3 clips |
| `code` | 1.7% | "get user by ID", "max retries", "parse JSON response": no identifier joined |
| `nouns-vocabulary` | 0.0% | "Zorvane, Kelthmar": a comma inside the name in 6 of 9 clips |
| `selfcorrection` | 0.0% | "to Sam, before lunch": a stray comma in 2 of 4 |
| a correction between amounts | not in the corpus | "$$50" scores as "fifty" |
So the numbers in `Docs/performance.md` cannot show a formatting regression or an improvement in these categories.
## Why it matters
The bench exists so a change to clean-up has something to compare against (#452). A pass that formats email addresses, or one that breaks them, would not move its numbers.
## Where to start
- `score()` in `Scripts/dictation_bench.py` already has `c["written"]` and `r["text"]`. Add a second column per category: the share of clips whose inserted text equals `written` after only whitespace and a trailing full stop are normalised (optionally ignoring case at the start).
- Keep the existing WER columns unchanged.
- Update the table header and the re-running section of `Docs/performance.md` to say what the new column means (the numbers themselves can wait for the next full run).
- No Swift change, so `python3 Scripts/dictation_bench.py score` on an existing `run.out` is enough to check; still run `make verify` before pushing. CONTRIBUTING.md has the rest.
## Acceptance criteria
- `score` prints an exact-text pass rate per category next to the WER.
- On a run like the one above, `emails` shows 0 of 3 and `nouns-vocabulary` shows the comma failures.
- `Docs/performance.md` explains the column.
**Size:** S, about 2 hours.
Contributor guide
Research direction
Start in score() in Scripts/dictation_bench.py, using the existing c["written"] and r["text"] values, then inspect the table and re-running section in Docs/performance.md. Run python3 Scripts/dictation_bench.py score on an existing run.out and verify that the new per-category exact-text rate exposes the stated failures while WER remains unchanged; finish with make verify.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, testing-qa
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 89/100