uttrflow / uttrflow/uttrflow-swift

dictation_bench.py scores email addresses, identifiers, symbols and punctuation as correct when they are not

Open Beginner friendly
#568 0 comments 0 reactions 0 assignees View on GitHub
area:dictation enhancement good first issue P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

`Scripts/dictation_bench.py` reports one metric, word error rate after `normalise()`, which lower-cases, drops all punctuation, rewrites `x@y` as "x at y", splits camelCase and `snake_case`, turns `/` into "slash" and a dot between letters into "dot". That is right for "were the words heard", but it makes several categories report 0% or near it while every clip in them is wrong in the inserted text:

| Category (bench on main) | Reported final WER | What was inserted |
|---|---|---|
| `emails` | 2.2% in `Docs/performance.md` | "support at example.com" instead of "support@example.com" in all 3 clips |
| `code` | 1.7% | "get user by ID", "max retries", "parse JSON response": no identifier joined |
| `nouns-vocabulary` | 0.0% | "Zorvane, Kelthmar": a comma inside the name in 6 of 9 clips |
| `selfcorrection` | 0.0% | "to Sam, before lunch": a stray comma in 2 of 4 |
| a correction between amounts | not in the corpus | "$$50" scores as "fifty" |

So the numbers in `Docs/performance.md` cannot show a formatting regression or an improvement in these categories.

## Why it matters

The bench exists so a change to clean-up has something to compare against (#452). A pass that formats email addresses, or one that breaks them, would not move its numbers.

## Where to start

- `score()` in `Scripts/dictation_bench.py` already has `c["written"]` and `r["text"]`. Add a second column per category: the share of clips whose inserted text equals `written` after only whitespace and a trailing full stop are normalised (optionally ignoring case at the start).
- Keep the existing WER columns unchanged.
- Update the table header and the re-running section of `Docs/performance.md` to say what the new column means (the numbers themselves can wait for the next full run).
- No Swift change, so `python3 Scripts/dictation_bench.py score` on an existing `run.out` is enough to check; still run `make verify` before pushing. CONTRIBUTING.md has the rest.

## Acceptance criteria

- `score` prints an exact-text pass rate per category next to the WER.
- On a run like the one above, `emails` shows 0 of 3 and `nouns-vocabulary` shows the comma failures.
- `Docs/performance.md` explains the column.

**Size:** S, about 2 hours.

Contributor guide

Open the contributing guide

Research direction

Start in score() in Scripts/dictation_bench.py, using the existing c["written"] and r["text"] values, then inspect the table and re-running section in Docs/performance.md. Run python3 Scripts/dictation_bench.py score on an existing run.out and verify that the new per-category exact-text rate exposes the stated failures while WER remains unchanged; finish with make verify.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, testing-qa
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
89/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.