uttrflow / uttrflow/uttrflow-swift

The evaluation scorer records a transcript in Arabic script as answered in Latin

Open
#495 0 comments 0 reactions 0 assignees View on GitHub
area:dictation bug good first issue P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

`Script.of` in `Sources/UttrflowEval/TextNormaliser.swift:11-13` only asks whether any Devanagari is present. Everything else is `.latin`. `TranscriptionScorer.score` (`Sources/UttrflowEval/TranscriptionScorer.swift:25`) stores that answer as `answeredIn`, so a transcript written in Arabic script (Unicode block U+0600-U+06FF) is recorded as `answeredIn: .latin` and then compared against the Latin reference.

## Why it matters

Hindi speech coming back in Urdu script is a known failure of the recogniser (see #378). The evaluation corpus is where that failure should be counted, and today it is reported as a Latin transcript with a very high word error rate instead of as a script problem, so the breakdown cannot see it.

## How to check

In `Tests/UttrflowEvalTests/TextNormaliserTests.swift` (the `Script` suite around line 111), `Script.of` on a string made only of Arabic-script letters returns `.latin`.

## Acceptance criteria

- `Script` gains a case for Arabic script, and `Script.of` returns it for text containing Arabic-script letters (Devanagari still wins if both appear, as it does today).
- `ReportBreakdown` notes an Arabic-script answer the way it notes `answeredInDevanagari` (`Sources/UttrflowEval/ReportBreakdown.swift:33, 43, 122`).
- `Script` is `Codable`: existing stored scores with `latin` and `devanagari` still decode.
- Tests cover Arabic script, mixed Arabic and Latin, and the existing cases.

## Where to start

- `Sources/UttrflowEval/TextNormaliser.swift:4-14`
- `Sources/UttrflowEval/ReportBreakdown.swift`
- Search for `.devanagari` across `Sources/UttrflowEval` to find every `switch` over `Script` that needs the new case.
- Tests to extend: `Tests/UttrflowEvalTests/TextNormaliserTests.swift`, `Tests/UttrflowEvalTests/ReportBreakdownTests.swift`, `Tests/UttrflowEvalTests/TranscriptionScorerTests.swift`.
- Before pushing, run `make verify` (export DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer first). It is the same command CI runs, and it enforces the 95% coverage floor per module.
- Read [CONTRIBUTING.md](https://github.com/uttrflow/uttrflow-swift/blob/main/CONTRIBUTING.md) first, and say on this issue that you are taking it. Adding a case to a public enum can leave stale build products; if tests fail with impossible results, run `swift package clean`.

**Size:** M, about half a day.

Contributor guide

Open the contributing guide

Research direction

Read CONTRIBUTING.md first, then inspect Script.of in Sources/UttrflowEval/TextNormaliser.swift and search for .devanagari across Sources/UttrflowEval. Extend the listed TextNormaliser, ReportBreakdown, and TranscriptionScorer tests for Arabic, mixed scripts, Codable compatibility, and existing cases. Run make verify with DEVELOPER_DIR set; done means the new breakdown behavior and tests pass without regressing stored latin or devanagari scores.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
Half a day
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.