SSD streaming: do these measurements from a separate MoE-offload prototype hold on V4 Flash? (eviction key, L+2 router lookahead, MTP union cost)

I spent a while building a separate expert-streaming prototype before I found this project, and ended up with a handful of measurements I cannot test against V4 Flash. Rather than let them rot, I wanted to ask whether any of them ring true here — especially the first two, which look like they might apply to the Metal streaming path.

Everything below was measured on **MiniMax-M2.5** (62 layers, 256 experts/layer, top-8, affine 4-bit hot / 2-bit cold under MLX) on a 24 GB M4, in Python. So the absolute numbers are not yours, and my per-layer overhead is roughly 30x yours. I am posting it because the *regime* looks nearly identical to #627: that report measures ~1.7 GiB/token on a 24 GB M5 with the cache holding a few percent of the routed set, and I measure 1.31–1.72 GB/token in the same shape. Everything here is self-contained, so you can judge it without following any links.

Genuine questions, not proposals. Please tell me if any of it is already known or already tried — 65k lines is a lot to have read carefully.

---

## 1. Is the eviction key ordering deliberate?

In `ds4_metal.m` (around :12574 and :12677) the victim is chosen by hotlist hotness first, with `last_used` only breaking ties inside a hotness class:

```c
(hotness < lowest_hotness ||
(hotness == lowest_hotness && e->last_used < oldest))
```

So the static compiled hotlist is the primary key, and recency is the tiebreak.

What made me look twice: I tried to find out how good *any* static policy could possibly be, before writing a smarter one. Method — take a real routing trace, count hits per expert, then solve the **fractional** knapsack over (hits ÷ bytes) at a fixed byte budget. Fractional, so it is a true upper bound rather than an achievable policy: no static trace-derived pinning can beat it.

That bound came out at **45.0% hit rate at a 9 GB budget**. Plain LRU, same trace, same budget, measured **53.0%**.

Which is why I dropped the idea of a smarter static policy entirely — the ceiling was below what recency was already doing for free.

Does that ordering hold on V4 Flash? Two reasons it might not: my trace is one prompt with short outputs, and greedy decode that falls into a loop flatters LRU; and your hotlist is profile-generated across a corpus, which is a strictly better estimator than the trace-derived pinning I bounded. But 8 points is a wide gap, and if it survives, the current primary key would be the weaker signal.

A smaller related one, if useful: splitting the cold half of the cache into probationary/protected (SLRU — a cold entry is only promoted into the protected half on an actual demand hit, so one-pass scans and bad prefetches cannot evict recurring entries) measured, at fixed ceiling over 32 decode tokens:

| ceiling | LRU | SLRU-cold |
|---|---:|---:|
| 8 GB | 48.56 GB read | 47.84 GB (2.00 → 2.03 t/s) |
| 10 GB | 40.53 GB read | 39.58 GB (2.29 → 2.32 t/s) |

Small, but it was free and it did not regress at any ceiling I tried.

## 2. Has cross-layer router lookahead been tried?

`AGENT.md` describes hiding misses behind the current layer's resident compute, and I could not find any `layer+1` router evaluation used to start reads earlier. So I do not know whether this was tried and rejected, or never tried.

I measured how far ahead the routing signal is actually usable — running layer L's router on the MoE input of an *earlier* layer, and asking how much of L's real top-8 it recovers:

| router evaluated at | recovers of layer L's top-k |
|---|---:|
| L+1 | 78.5% |
| **L+2** | **72.5%** |
| L+3 | 68.0% |
| same layer, next token | 33.0% |

That last row is the obvious prefetch everyone tries first, and it is why it fails.

I then implemented the L+2 version, and **it lost**: 2.22 t/s against 2.34. But it lost for a reason that is specific to my runtime, not to the idea — hit rate went 75.3% → 76.7% and time blocked on disk went 10.8s → 10.4s, exactly as designed. What killed it is that time *off* the disk grew 6.5s → 7.7s: 1984 extra router calls at ~0.6 ms each, where the 0.6 ms is a Python matmul, an argsort, and a `.tolist()` that forces a GPU sync. It spent 0.6 ms of interpreter per layer to save 0.2 ms of disk.

In C that arithmetic looks like it should invert — a router pass is a small matvec and a partial sort. And it would buy two full layers of lead time rather than one layer's overlap, which seems to matter most on exactly the machines in #627 where decode is SSD-bandwidth-bound.

Is there a reason this does not work here that I am not seeing?

## 3. On #596 (`--ssd-streaming` + `--mtp`): does the union cost match?

Speculation is close to free on a resident model, because verifying γ tokens costs one weight read regardless of γ. Streaming breaks that assumption: a verify pass has to read the **union** of its tokens' routed experts, and rejected drafts pay for theirs too.

I measured that union over 64 decode tokens:

| γ | experts in the union (top-8 model) |
|---:|---:|
| 1 | 8.00 |
| 2 | 13.42 |
| 3 | 17.99 |
| 8 | 35.01 |

Roughly `k · (1 + 0.58·(γ−1))` — the union grows at a bit over half the independent rate, because consecutive tokens share routing (33.4% of a token's top-8 is shared with the previous token).

Turning that into a bound on end-to-end speedup:

- **1.83x** ceiling with a draft that is never wrong
- 1.30x at 90% acceptance
- 1.08x at 80%
- **net loss below ~72% acceptance**

And that is generous, because the union is measured along the trace the model actually produced — a rejected draft routes somewhere else, so the real union is wider.

If the same slope holds on V4 Flash, the feature would hinge entirely on measured MTP acceptance, and the payoff would cap near 1.8x rather than the 2–3x that GPU-resident speculation gets. Does that match what you have seen? I ask because if acceptance is known already, that one number seems to decide the whole feature.

## 4. Three things I tried that did not work

Posting these mostly so nobody repeats them. Each was killed by a measurement, not by intuition:

- **Factoring experts into a shared basis plus per-expert residual.** If `W_e = B + Δ_e` with `B` resident, only the residual streams. Dead: mean pairwise cosine between a layer's experts is **+0.096**, mean-subtraction leaves 98.6% of the norm, and capturing 83% of the energy takes rank 128 of 256 — a basis costing more than what it factors. The experts are near-orthogonal.
- **Adaptive top-k / skipping experts the cache does not hold.** Very tempting, because a resident expert is free and a missing one is a stall, so the quality loss lands exactly where the speed gain is. Dead on the gate distribution: **0.192 at slot 1 against 0.093 at slot 8**. Sigmoid top-k is flat, unlike the skewed softmax top-2 that the adaptive-k literature assumes. Dropping *one* expert cost 11.4% block error.
- **Using the resident cache as its own draft model** — skip the experts you do not have, renormalise, draft, verify the batch against disk. No second model and no extra memory, and it improves as the cache warms. Drafts at 189 ms/token against 1053, **5.6x** — at **16.7% acceptance**. Downstream of the flat gates above.

## 5. A read-path datapoint, probably not actionable

You use buffered `pread` plus `MADV_WILLNEED`/`MADV_DONTNEED`, which on a unified-memory Mac seems right to me — I found the OS page cache genuinely useful as a reclaimable second-level cache underneath a hard arena.

One number anyway, since `[MTLBuffer contents]` hands out a 16 KB-aligned writable pointer and MLX does the same:

| path | throughput, 1.18 MB blocks |
|---|---:|
| `pread` → heap → GPU array | 1.89 GB/s |
| `preadv` → freshly allocated GPU buffer | 1.49 GB/s |
| `preadv` with `F_NOCACHE` → **preallocated** GPU-visible slot | **2.62 GB/s** |

The middle row is the interesting one: allocating per read is worse than the copy it saves, because zeroing writes the bytes the read is about to overwrite.

#627 reports ~4.4 GB/s through your path, so you are likely at the device limit already and this is worth nothing on bandwidth. The only argument I would make is about control rather than speed: on a 24 GB machine the page cache and the expert cache are the same RAM, and `F_NOCACHE` is what makes that split explicit instead of emergent.

---

I do not have a V4 Flash setup and cannot run your regression suite, so I am asking rather than proposing — and I am aware that (1) and (2) are cheap for me to suggest and not cheap for you to test. If any of it is worth pursuing I am happy to describe the methods in more detail; if it is all already known, no reply needed and thanks for the engine.