uttrflow / uttrflow/uttrflow-swift
Scoring a suggestion tokenises the candidate twice, the largest tokenizer cost left after #449
- Dominant language
- Swift
- Stars
- 4
- Forks
- 17
- Avg merge
- 3h 32m
- Merged PRs (30d)
- 277
Description
## What happens
`MLXCandidateScorer.judge` (`Sources/UttrflowLocalModel/MLXCandidateScorer.swift:620-639`) encodes the whole line and then encodes its typed opening separately:
```swift
let whole = loaded.tokenizer.encode(text: leadIn + candidate)
let typed = loaded.tokenizer.encode(text: leadIn + typedPart(of: candidate, following: context))
```
(lines 623-624), only to find where the two token streams diverge. `Docs/performance.md` (around line 407) measures this at about 100 ms of processor time per pass, now the largest tokenizer cost since #449 cached the prompt's frame and lines. Most of it is the dependency's slow regex for added tokens (tracked upstream in #489).
## Why it matters
A score runs on every suggestion pass, which follows typing. 100 ms of processor a pass is battery spent on work that can mostly be reused.
## How to check
Read lines 620-639. The `gpu-memory --typing` harness reports processor time per pass (`make bakeoff ARGS="gpu-memory --typing"`).
## Acceptance criteria
- The typed opening is not tokenised from scratch on every pass. Options: tokenise the typed opening once per typed prefix and reuse it across the candidates of a pass; or derive the divergence point from one encoding where the tokenizer's offsets allow it. `firstScoredIndex` must still be right when the join retokenises (its existing tests must pass).
- Scores are identical before and after on the bake-off's scoring fixtures.
- Processor time per pass is measured before and after with `gpu-memory --typing`, and `Docs/performance.md` is updated.
Contributor guide
Research direction
Read MLXCandidateScorer.swift lines 620-639 and Docs/performance.md around line 407. Run make bakeoff ARGS="gpu-memory --typing" to establish processor time, then verify existing firstScoredIndex tests and bake-off scoring fixtures. Done means the typed opening is reused, scores remain identical, and Docs/performance.md records before-and-after measurements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100