uttrflow / uttrflow/uttrflow-swift

Accuracy baseline comparison accepts a different engine, model or language-hint configuration

Open
#823 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## Problem

`AccuracyBaseline.compare(with:)` never compares the baseline label with the new report label. Its `incomparability` check considers shared sample IDs and normalization rules only. Consequently a baseline from one recognizer configuration can produce an ordinary unchanged/improved/worsened verdict for another configuration.

This contradicts both the `AccuracyBaseline.label` contract and `Docs/eval-methodology.md` (Regression tolerance), which require different engine/model/hinting labels to be incomparable. The command passes a user-selected baseline straight to this comparison; its label includes engine, model variant and language hinting.

## Verified evidence

Reviewed main commit `5c310667cd35c44abd8bbd26c19b0905471287b4`:

- `Sources/UttrflowEval/AccuracyBaseline.swift`: `compare(with:)` and `incomparability(with:shared:)` do not check `label != report.label`.
- `Sources/uttrflow-eval/TranscribeCorpus.swift`: `compare(reporting:)` reads the selected baseline and calls that comparison without a configuration check; `label(_:)` distinguishes engine/model/hinting.
- A controlled Swift test calling the unchanged production scorer/report types captured a baseline labeled `whisperKit tiny, language detected`, then compared an identical synthetic passage in a report labeled `appleSpeech tiny, language detected`. Actual result: `verdict=unchanged`, `reason=nil`; the expectation of `.incomparable` fails.

The probe tests comparison logic, not actual speech-engine accuracy. The existing 28 baseline/store tests pass, but none tests a changed label.

## Impact and acceptance criteria

Selecting the wrong baseline file can yield a misleading regression verdict instead of identifying that the runs describe different configurations.

- Reject unequal configuration labels with an explicit incomparable reason before presenting a regression verdict.
- Add coverage for engine, model and hinting changes, with same-configuration comparisons retained as controls.
- Preserve shared-sample comparison and existing normalization checks.

Whether the CLI should fail on every incomparable result is a separate policy question; this issue concerns the missing configuration check.

Contributor guide

Open the contributing guide

Research direction

Start in Sources/UttrflowEval/AccuracyBaseline.swift, then read Sources/uttrflow-eval/TranscribeCorpus.swift and the existing 28 baseline/store tests. Run the baseline tests first and trace compare(with:) through incomparability(with:shared:). Done means engine, model, and hinting label changes produce an explicit incomparable result, while same-label, shared-sample, and normalization cases retain their existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.