uttrflow / uttrflow/uttrflow-swift
Accuracy baseline comparison accepts a different engine, model or language-hint configuration
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## Problem
`AccuracyBaseline.compare(with:)` never compares the baseline label with the new report label. Its `incomparability` check considers shared sample IDs and normalization rules only. Consequently a baseline from one recognizer configuration can produce an ordinary unchanged/improved/worsened verdict for another configuration.
This contradicts both the `AccuracyBaseline.label` contract and `Docs/eval-methodology.md` (Regression tolerance), which require different engine/model/hinting labels to be incomparable. The command passes a user-selected baseline straight to this comparison; its label includes engine, model variant and language hinting.
## Verified evidence
Reviewed main commit `5c310667cd35c44abd8bbd26c19b0905471287b4`:
- `Sources/UttrflowEval/AccuracyBaseline.swift`: `compare(with:)` and `incomparability(with:shared:)` do not check `label != report.label`.
- `Sources/uttrflow-eval/TranscribeCorpus.swift`: `compare(reporting:)` reads the selected baseline and calls that comparison without a configuration check; `label(_:)` distinguishes engine/model/hinting.
- A controlled Swift test calling the unchanged production scorer/report types captured a baseline labeled `whisperKit tiny, language detected`, then compared an identical synthetic passage in a report labeled `appleSpeech tiny, language detected`. Actual result: `verdict=unchanged`, `reason=nil`; the expectation of `.incomparable` fails.
The probe tests comparison logic, not actual speech-engine accuracy. The existing 28 baseline/store tests pass, but none tests a changed label.
## Impact and acceptance criteria
Selecting the wrong baseline file can yield a misleading regression verdict instead of identifying that the runs describe different configurations.
- Reject unequal configuration labels with an explicit incomparable reason before presenting a regression verdict.
- Add coverage for engine, model and hinting changes, with same-configuration comparisons retained as controls.
- Preserve shared-sample comparison and existing normalization checks.
Whether the CLI should fail on every incomparable result is a separate policy question; this issue concerns the missing configuration check.
Contributor guide
Research direction
Start in Sources/UttrflowEval/AccuracyBaseline.swift, then read Sources/uttrflow-eval/TranscribeCorpus.swift and the existing 28 baseline/store tests. Run the baseline tests first and trace compare(with:) through incomparability(with:shared:). Done means engine, model, and hinting label changes produce an explicit incomparable result, while same-label, shared-sample, and normalization cases retain their existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100