uttrflow / uttrflow/uttrflow-swift

Docs/bakeoff.md's Marcy/Sara/Jon rule and two failing scores predate the doubtful-run fix

Open
#286 1 comment 0 reactions 0 assignees View on GitHub
area:dictation claimed documentation help wanted P2
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

`Docs/bakeoff.md` states a rule about when the clean-up model takes a spelling off the
screen, and draws it from four runs of `slack-name-spelling`. Two of those cases could not
reach the candidate path when the runs were taken, and the page now says so itself — but
only in one of the four places the numbers are used, and the rule is still written as
though it described the shipping pipeline.

## Where it is

`Docs/bakeoff.md:169-173`, added after the corpus was corrected:

> **These rows were taken with the candidate path switched off.** `slack-name-spelling`
> declared no doubtful run until the corpus was corrected, and a case that declares none is
> never asked of a candidate source — so the model was never handed `Marcie` and the table
> measures it noticing a spelling unaided. The case declares its run now, and the rule below
> is what the old runs suggested rather than what the shipping path does; it needs re-measuring.

Three places still carry the old numbers or the conclusion drawn from them:

- `Docs/bakeoff.md:145-153` — "Shipping fails five of thirty-six", with `slack-name-spelling` at 91% and `editor-selected-identifier` at 85%. No note here that either predates the fix.
- `Docs/bakeoff.md:161-167` — the four-row table (`Nikhil ✅`, `Marcy ❌`, `Sara ❌`, `Jon ❌`).
- `Docs/bakeoff.md:175-181` — the rule itself: the model takes a spelling only when the one it heard is "not itself a plausible name", and "there is no wording found so far that fixes it without also breaking the other two".
- `Docs/cleanup.md:129-139` names the same two among seven failing cases and carries the caveat already, so that page agrees with `:169-173` — but its count of seven and its list are still the ones from the old runs.

Both cases declare a doubtful run now — `Sources/UttrflowEval/EvaluationCorpus.swift:391`
(`doubtful: ["marcy"]`) and `:448` (`doubtful: ["set user prefs"]`) — and that is what
makes the difference: `Sources/UttrflowEval/EvaluationCase.swift:82-90` builds per-word
confidences only when `doubtful` is non-empty, and the same `transcription` is what
`transformationRequest` hands the engine, which the bake-off uses at
`Sources/uttrflow-bakeoff/Bakeoff.swift:183`. So a run today exercises a different pipeline
from the one that produced the table.

## What to change

Two steps, and the first does not need the model.

1. Mark the stale numbers where they are quoted. `:145-153` is the table a reader meets
first and it says nothing; the two rows should carry the same caveat `:169-173` does.

2. Re-run the two cases and replace the numbers and the rule with what the shipping path
actually does. The whole corpus:

```bash
make bakeoff
```

or one row at a time, per `Docs/bakeoff.md:183-191`, remembering the flag that page adds
for a case with a doubtful run:

```bash
uttrflow-dev clean -e foundationModels "thanks marcy i'll pick up the printer quote this afternoon" \
--app Slack --bundle-id com.tinyspeck.slackmacgap \
--document "Marcie Alvarez (DM) — Northwind" --doubtful "marcy"
```

Either outcome is a result worth having, and the page should say which it is and on what
date:

- The model now writes `Marcie` when handed the spelling. Then `:175-181`'s rule is a
description of the old path and has to be rewritten, and `Docs/cleanup.md`'s seven
failing cases are six or fewer.
- It still writes `Marcy`. Then the rule stands and is stronger than before, because the
model refused a candidate it was offered by name rather than one it had to notice.

Do not change the prompt to make a row pass — `Docs/cleanup.md:138-139` records that these
failures are kept honest rather than papered over, and a wording change that fixes `Marcy`
while breaking `Sara` and `Jon` has already been looked for and not found. This is a
measurement.

## How to check it

Every number written into either page comes from the run's own output, pasted or
transcribed, with the date. The pairing the run depends on is already pinned by
`Tests/UttrflowEvalTests/DoubtfulCorpusTests.swift:36-54`, which asserts that
`slack-name-spelling` offers `marcy` → `Marcie` and `editor-selected-identifier` offers
`set user prefs` → `setUserPrefs`, so if those tests pass the candidate path is reaching
the model:

```bash
swift test --filter DoubtfulCorpusTests
```

## Notes

This needs Apple's on-device model for `-e foundationModels`, and `make bakeoff`
downloads models and wants the Metal toolchain (`Makefile:171`) — which is why it is not
labelled a first issue. Step 1 alone is a documentation edit and can be done without any
of that.

The corpus fix this follows from is #148.

Contributor guide

Open the contributing guide

Research direction

Start with Docs/bakeoff.md:145-191 and Docs/cleanup.md:129-139, then inspect the doubtful-run handling in Sources/UttrflowEval/EvaluationCase.swift and Sources/uttrflow-bakeoff/Bakeoff.swift. Run swift test --filter DoubtfulCorpusTests, followed by make bakeoff or the documented case command with the doubtful flag. Done means both documentation pages mark stale figures, record dated current outputs, and state the rule and failure count supported by the shipping-path runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.