uttrflow / uttrflow/uttrflow-swift

Scoring a suggestion tokenises the candidate twice, the largest tokenizer cost left after #449

Open
#478 0 comments 0 reactions 0 assignees View on GitHub
area:ai-suggestions enhancement P2 performance
Dominant language
Swift
Stars
4
Forks
17
Avg merge
3h 32m
Merged PRs (30d)
277

Description

## What happens

`MLXCandidateScorer.judge` (`Sources/UttrflowLocalModel/MLXCandidateScorer.swift:620-639`) encodes the whole line and then encodes its typed opening separately:

```swift
let whole = loaded.tokenizer.encode(text: leadIn + candidate)
let typed = loaded.tokenizer.encode(text: leadIn + typedPart(of: candidate, following: context))
```

(lines 623-624), only to find where the two token streams diverge. `Docs/performance.md` (around line 407) measures this at about 100 ms of processor time per pass, now the largest tokenizer cost since #449 cached the prompt's frame and lines. Most of it is the dependency's slow regex for added tokens (tracked upstream in #489).

## Why it matters

A score runs on every suggestion pass, which follows typing. 100 ms of processor a pass is battery spent on work that can mostly be reused.

## How to check

Read lines 620-639. The `gpu-memory --typing` harness reports processor time per pass (`make bakeoff ARGS="gpu-memory --typing"`).

## Acceptance criteria

- The typed opening is not tokenised from scratch on every pass. Options: tokenise the typed opening once per typed prefix and reuse it across the candidates of a pass; or derive the divergence point from one encoding where the tokenizer's offsets allow it. `firstScoredIndex` must still be right when the join retokenises (its existing tests must pass).
- Scores are identical before and after on the bake-off's scoring fixtures.
- Processor time per pass is measured before and after with `gpu-memory --typing`, and `Docs/performance.md` is updated.

Contributor guide

Open the contributing guide

Research direction

Read MLXCandidateScorer.swift lines 620-639 and Docs/performance.md around line 407. Run make bakeoff ARGS="gpu-memory --typing" to establish processor time, then verify existing firstScoredIndex tests and bake-off scoring fixtures. Done means the typed opening is reused, scores remain identical, and Docs/performance.md records before-and-after measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.