uttrflow / uttrflow/uttrflow-swift

The AI suggestion fixture report counts any prose continuation as a hit and 137 designed URL refusals as misses

Open
#540 0 comments 0 reactions 0 assignees View on GitHub
area:ai-suggestions enhancement P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

The fixture report's hit rate and "precision" count AI suggestion outputs as right that plainly are not. They also count 137 designed refusals as misses. The headline numbers do not measure what they claim.

**1. Any continuation counts as a hit for most prose.** `CompletionExpectation.hits` returns true for any non-empty completion when `acceptable` is empty (`Sources/UttrflowEval/CompletionCase.swift:23-28`). That is `Determinacy.any`, the default for every chat scenario (`Sources/uttrflow-bakeoff/CatalogueChat.swift:144`), the notes release, report and to-do scenarios (`CatalogueNotes.swift:21`, `:53`, `:88`) and mail (`CatalogueMail.swift:58`), except lines individually marked `.prose`. From the run on `f969b41` (release, gemma3), all scored ✓ hits:

- `chat/group/who-is-picking/cut10`: `who is pic` → `who is pic pic pic pic pic`
- `notes/todo/pay-electricity-bill/cut3`: `- p` → `- p a s s p o r t`
- `notes/todo/cancel-gym-membership/cut5`: `- can` → `- can i check my balance`
- `chat/work/standup-notes-are/cut10`: `Standup no` → `Standup no problem.`
- `robust/long/the-evaluation-set/cut200`: `…and a thousand genera` → `…genera of birds do not have a single, consistent way to perceive the world.`

The summary line was `precision 96.44 % (947/982 shown, 35 wrong)`, and chat scored `hit 293/298`. Neither figure can see these.

**2. URL and search fixtures expect answers the product refuses by design.** `Register.answersFromHistoryAlone` makes every pass in an address or search field return nothing (`Sources/UttrflowPredict/Register.swift:75`, `Sources/UttrflowLocalModel/MLXCandidateScorer.swift:308`). Yet the `url/` fixtures still expect completions (`Sources/uttrflow-bakeoff/Fixtures.swift:166-167`, `CatalogueURL.swift`). This run: `url hit 0/137`, `search hit 0/1`. That is 138 of the 189 misses in `all hit 965/1154`, none of them a model failure.

## Why it matters

Decisions about prompts, token healing and model choice are taken from these numbers (`Docs/predict-reliability.md`). A judge that passes `pic pic pic pic pic` cannot catch a regression in prose, which is most of what people type. Unrefusable misses hide real movement in the other 1,016 fixtures.

## Acceptance criteria

- `.any` fixtures get a minimal plausibility check before counting as a hit: no degenerate repetition (#538), the fragment not split into a non-word (#539), no invented tokens for mid-word cuts. Or they are reported in a separate "unjudged" bucket that is excluded from precision.
- `url/` and `search/` fixtures expect `` (or are reported under their own heading) while `answersFromHistoryAlone` is the design.
- The report prints how many hits were judged against a concrete expectation vs. accepted unjudged.
- `Docs/predict-reliability.md` records the rescored baseline.

Contributor guide

Open the contributing guide

Research direction

Start with CompletionCase.swift, CatalogueChat.swift, CatalogueNotes.swift, CatalogueMail.swift, Register.swift, MLXCandidateScorer.swift, and the URL fixtures to trace how expectations become report results. Compare the concrete-expectation, .any, URL, and search paths, then run the fixture report against the cited baseline. Done means unjudged or plausibility-checked results are separated, refusals are not counted as model misses, the report shows both hit categories, and Docs/predict-reliability.md records the rescored baseline.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.