uttrflow / uttrflow/uttrflow-swift

With Suggestions on, a 5,000-pass soak goes over the memory budget: 4.28 GB peak and 3.08 GB between passes

Open
#475 1 comment 0 reactions 0 assignees View on GitHub
area:ai-suggestions bug P2 performance
Dominant language
Swift
Stars
4
Forks
17
Avg merge
4h 27m
Merged PRs (30d)
286

Description

## What happens

`Docs/performance.md` (the memory budget table, lines 226-234) sets two lines for Suggestions on: at most 3.0 GB between passes and at most 3.5 GB at the peak of a pass. A 5,000-pass soak of the suggestion model on current `main`, after the reload leak fixed in #462, measured:

| reading | budget | measured |
|---|---|---|
| process peak | 3.5 GB | about 4.28 GB |
| footprint between passes | 3.0 GB | about 3.08 GB |

The peak is well over the line; the between-passes figure is just over it. #462 measured 4.52 GB and 3.39 GB in the same kind of run before its fix, so the fix helped but did not bring either reading under the budget.

## Why it matters

The budget is written for an 8 GB Mac, where the suggestion model is already close to half the memory. A peak 0.8 GB over the line is the difference between fitting and pushing the user's other apps into swap.

## How to reproduce

The soak harness used in #462 is not in the repository. The closest committed equivalent is:

```bash
make bakeoff ARGS="gpu-memory --passes 5000 --release"
```

which checks every pass's peak and settled footprint against `ResourceBudget` and exits non-zero on a breach. `make perf-budget-models` runs only 12 passes, which is why it passes today. Run with `/usr/bin/time -l` to read the process peak as well.

## Acceptance criteria

- The source of the growth over a long run is identified: MLX's cache, the key-value cache, allocator fragmentation, or the remaining first-load cycles tracked in #411.
- A 5,000-pass run stays under both lines, or `Docs/performance.md` records a measured, justified change to the budget.
- The soak is runnable from the repository (a `gpu-memory` option or a documented command), so the next regression is found by running it rather than by rebuilding a harness.

Contributor guide

Open the contributing guide

Research direction

Start with Docs/performance.md lines 226-234, ResourceBudget, and the gpu-memory path used by `make bakeoff ARGS="gpu-memory --passes 5000 --release"`; compare it with `make perf-budget-models`. Trace the long-run growth against the possibilities listed in the issue and #411/#462. Done means a repository-runnable 5,000-pass soak stays within both budgets, or the documentation records a measured, justified budget change.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.