sgl-project / sgl-project/sglang
[Prefix Cache / Hybrid] Branch reuse silently collapses under LRU pressure when (kv_pool_tokens / chunked_prefill_size) > max_mamba_cache_size
- Dominant language
- Python
- Stars
- 36.1k
- Forks
- 9k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 219
Description
### Checklist
- [x] I searched related issues but found no solution.
- [x] The bug persists in the latest version.
- [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
- [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
- [x] Please use English. Otherwise, it will be closed.
I've been characterizing Qwen3.8-27B-FP8 serving on a single RTX PRO 6000 (sm_120, 96 GB, TP=1, `lmsysorg/sglang:v0.5.17-cu130`, `--enable-hierarchical-cache --hicache-size 150 --hicache-write-policy write_through`) under agentic workloads, and ran into a failure mode that I could eventually reproduce deterministically and turn on and off with single flags. Reporting it because #22326 anticipated exactly this capacity question and explicitly left it open ("assumes older checkpoints evict quickly when newer requests arrive, but this isn't guaranteed").
The observation: requests that share a prefix with a cached sequence but diverge mid-way (context compaction rewrites, session branches — the normal shape of agent traffic) go from a ~40k-token prefix hit to a 0-token hit, with no error and no log signal. Exact repeats keep hitting, so the regression is invisible unless you watch branch requests specifically. TTFT for the affected requests jumps from ~0.3 s to full cold prefill (~15 s at 80k in my probe; ~55 s at 190k).
Mechanism, as far as I can tell from reading the code and confirming with counters: interior mamba checkpoints are only created at chunked-prefill boundaries (`cache_unfinished_req` inserts once per chunk and donates the state), so checkpoint spacing along a cached chain equals `chunked_prefill_size`. A branch match needs the checkpoint at the divergence point, and KV+mamba are gated jointly — KV can be 100% resident and the match is still rejected if the mamba checkpoint is gone. On match, only the last node's mamba state is MRU-refreshed (`mamba_lru_list.reset_node_mru(last_node)`), so interior checkpoints age out under any sustained load. That gives a capacity criterion:
```
checkpoint demand ≈ kv_pool_tokens / chunked_prefill_size
demand > max_mamba_cache_size => branch reuse collapses under LRU pressure
```
I tested the criterion from both sides on one box. Probe: build six distinct 80k-token chains, then issue a fresh branch request forking the oldest chain at its midpoint (~40k). `mamba_used/available/evictable` gauges from `/metrics` conserve to the pool size throughout, which is how I verified saturation.
| config | slots | kv_pool/cps | branch hit | pool at probe |
|---|---|---|---|---|
| cps=8192, fp32 | 165 | 433,919/8192 = 53 | 32,768 | evictable peak 44 |
| cps=2048, fp32 | 165 | 433,919/2048 = 212 | **0** | saturated, avail=5 |
| cps=2048, `--mamba-track-interval 8192` | 165 | 212 | 0 (byte-identical run) | saturated |
| cps=2048, `--mamba-ssm-dtype bfloat16` | 325 | 212 | 32,768 | avail ≥111 |
Three things I'd flag from this:
1. The collapse follows the criterion, not the workload. Doubling the state pool (bf16) with everything else fixed makes it disappear; halving checkpoint spacing (cps 8192→2048) with everything else fixed makes it appear. Hit lengths are exactly cps-quantized throughout (32,768 = 16×2048, 38,912 = 19×2048), which also confirms the spacing claim.
2. `--mamba-track-interval` does not participate: setting it to 8192 changed nothing, run-for-run identical. As documented it's decode-side only. So today the *only* knob controlling prefill checkpoint density is `chunked_prefill_size` — a knob operators tune for prefill/decode interference. Lowering cps to protect TBT under concurrency silently pushes the engine across the criterion. That coupling is the actual footgun, and it's why I think the `prefill_cache_interval` idea from #22326 deserves to be revived: it's the missing independent knob.
3. At saturation the damage is broader than branches: cold chains lose their tail states too, so even exact-repeat requests for not-recently-touched sequences drop to 0 (observed on 2 of 5 chains in the saturated runs). The usual "exact repeats always hit" intuition only holds while the pool has headroom.
For what it's worth, this isn't a synthetic-only concern: serving 10 concurrent agent sessions (SWE-bench-style, 30–200k contexts) with cps=2048 on the same hardware, I've measured the mamba pool at 0.84–0.97 usage in steady state — the criterion regime is where agentic serving actually lives. Also possibly relevant to the `mamba_full_memory_ratio` discussion in #25993: with hicache enabled the derived pool on this box drops 165→147, and with EAGLE on top it drops to 70, so the criterion tightens twice before any user tuning happens.
Concrete asks, in increasing order of effort:
1. **A startup warning when the criterion is crossed.** At boot the engine already logs how the mamba pool caps `max_running_requests`; one more line when `kv_pool_tokens / chunked_prefill_size > max_mamba_cache_size` would name the coupling before an operator discovers it via TTFT spikes. A few lines, no behavior change — I can send this PR if you want it.
2. **A counter for mamba-gated match rejections.** When a branch match is rejected because the checkpoint is missing (KV present, mamba absent), nothing distinguishes it from a genuinely cold request today. A metric (or a debug-level log) would make the failure mode observable in production.
3. **Revive `prefill_cache_interval` from #22326.** Checkpoint density should have its own knob instead of riding on `chunked_prefill_size`; the capacity criterion above is the sizing rule that proposal said was missing ("assumes older checkpoints evict quickly … this isn't guaranteed" — measured: it isn't, and what gets evicted is exactly what branch reuse needs). With an independent interval, operators can tune cps for prefill/decode interference and checkpoint density for reuse, separately.
Happy to share the probe (~150-line standalone script against `/generate` + `/metrics`) and raw per-request JSONL for all four configs, or to rerun with different settings if that helps. I'm aware this is one box, one model family, synthetic chains; the numbers above are per-config single runs, but the on/off behavior reproduced every time I ran it.
Contributor guide
Assessment
This issue has not been assessed yet.