uttrflow / uttrflow/uttrflow-swift
Self-correction by restatement is asked of the model in the prompt contract and the model does not do it
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
The clean-up catalogue asks for one thing that nothing in the app actually does. A slot said
twice with a different word after it — "as a gift as a present" — should keep the later one.
The rules deliberately do not attempt it, the model is asked to, and a measurement on
2026-09-06 found the model does not comply. So the words stay, which is the safe side of the
line, but the catalogue currently claims a behaviour the product does not have.
## Where it is
`Sources/UttrflowAI/PromptContract.swift:14` asks for it, in the contract every dictation is
tidied under:
```
- a slot said twice keeps the later: "as a gift as a present"
```
and line 50 shows it as one of the worked examples the model is given.
`Docs/cleanup.md:71` records the measurement and the reasoning:
> **Measured 2026-09-06: the model does not comply.** "I wanted to buy a record as a gift as a
> present" and "let's meet on tuesday on wednesday afternoon" both come back whole.
The same row explains why a deterministic rule is refused, and it is worth reading before
proposing one: the shape "a short frame of function words repeated with a different content
word after each copy" matches "I'll pay for lunch for everyone" and "coffee with milk with
sugar" just as often as it matches a real self-correction. Which one the speaker meant to
stand is semantic. **A pull request that adds that rule will not be merged** — this is settled
and the reasoning is in the doc.
## What to change
The useful work is measurement, not a fix:
1. **Add corpus cases** to `Sources/UttrflowEval/EvaluationCorpus.swift` for restatement, in
the shape of the existing `grammar` block at line 769 — an `EvaluationCase` with `spoken`,
`expected`, `mustKeep` and `mustNotAdd`. Two exist in the doc already and more phrasings
would help: how far apart the two halves can be, whether a trigger phrase ("no, sorry",
"I mean") changes the answer, whether it behaves differently in a message and a document.
2. **Run them and report what the model does**, so this stops being one observation on two
sentences. A case that fails is the point — it makes the gap visible and any future prompt
change measurable.
3. If a *prompt* change moves the number, that is worth a pull request; say what you measured
before and after.
## How to check it
The evaluation harness runs the real pipeline, so this needs an Apple Silicon Mac with the
model installed (`CONTRIBUTING.md` has the setup, and `Docs/eval-methodology.md` explains how a
case is scored). Report the model and the macOS version with any number, and give the exact
sentences — a percentage with no sentences behind it cannot be checked by anybody.
## Notes
- Do not "fix" this by deleting the line from the contract. The instruction is cheap and the
behaviour is wanted; what is missing is evidence about when the model honours it.
- `Docs/cleanup.md:71` is the specification. If the measurement changes the picture, that row
changes with it in the same pull request.
Contributor guide
Research direction
Start with the grammar block around line 769 in Sources/UttrflowEval/EvaluationCorpus.swift, then read CONTRIBUTING.md and Docs/eval-methodology.md before running the evaluation harness on an Apple Silicon Mac. Add restatement cases with spoken, expected, mustKeep, and mustNotAdd, and report exact sentences, model, macOS version, and results; update Docs/cleanup.md:71 if the measurement changes the picture.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, swift
- Domain
- documentation, testing
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100