uttrflow / uttrflow/uttrflow-swift
The evaluation scorer records a transcript in Arabic script as answered in Latin
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## What happens
`Script.of` in `Sources/UttrflowEval/TextNormaliser.swift:11-13` only asks whether any Devanagari is present. Everything else is `.latin`. `TranscriptionScorer.score` (`Sources/UttrflowEval/TranscriptionScorer.swift:25`) stores that answer as `answeredIn`, so a transcript written in Arabic script (Unicode block U+0600-U+06FF) is recorded as `answeredIn: .latin` and then compared against the Latin reference.
## Why it matters
Hindi speech coming back in Urdu script is a known failure of the recogniser (see #378). The evaluation corpus is where that failure should be counted, and today it is reported as a Latin transcript with a very high word error rate instead of as a script problem, so the breakdown cannot see it.
## How to check
In `Tests/UttrflowEvalTests/TextNormaliserTests.swift` (the `Script` suite around line 111), `Script.of` on a string made only of Arabic-script letters returns `.latin`.
## Acceptance criteria
- `Script` gains a case for Arabic script, and `Script.of` returns it for text containing Arabic-script letters (Devanagari still wins if both appear, as it does today).
- `ReportBreakdown` notes an Arabic-script answer the way it notes `answeredInDevanagari` (`Sources/UttrflowEval/ReportBreakdown.swift:33, 43, 122`).
- `Script` is `Codable`: existing stored scores with `latin` and `devanagari` still decode.
- Tests cover Arabic script, mixed Arabic and Latin, and the existing cases.
## Where to start
- `Sources/UttrflowEval/TextNormaliser.swift:4-14`
- `Sources/UttrflowEval/ReportBreakdown.swift`
- Search for `.devanagari` across `Sources/UttrflowEval` to find every `switch` over `Script` that needs the new case.
- Tests to extend: `Tests/UttrflowEvalTests/TextNormaliserTests.swift`, `Tests/UttrflowEvalTests/ReportBreakdownTests.swift`, `Tests/UttrflowEvalTests/TranscriptionScorerTests.swift`.
- Before pushing, run `make verify` (export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer first). It is the same command CI runs, and it enforces the 95% coverage floor per module.
- Read [CONTRIBUTING.md](https://github.com/uttrflow/uttrflow-swift/blob/main/CONTRIBUTING.md) first, and say on this issue that you are taking it. Adding a case to a public enum can leave stale build products; if tests fail with impossible results, run `swift package clean`.
**Size:** M, about half a day.
Contributor guide
Research direction
Read CONTRIBUTING.md first, then inspect Script.of in Sources/UttrflowEval/TextNormaliser.swift and search for .devanagari across Sources/UttrflowEval. Extend the listed TextNormaliser, ReportBreakdown, and TranscriptionScorer tests for Arabic, mixed scripts, Codable compatibility, and existing cases. Run make verify with DEVELOPER_DIR set; done means the new breakdown behavior and tests pass without regressing stored latin or devanagari scores.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- Half a day
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100