Server: multimodal sessions lose all KV reuse on a tail token mismatch — no common-prefix rewind, no disk-KV fallback, no continued disk stores
- Ngôn ngữ chính
- C
- Star
- 22.3k
- Fork
- 2.1k
- Merge trung bình
- 1 ngày 3 giờ
- Pull request đã merge (30 ngày)
- 4
Mô tả
**Title:** Server: multimodal sessions lose all KV reuse on a tail token mismatch — no common-prefix rewind, no disk-KV fallback, no continued disk stores
**Observed on:** `6289c51` (current main), DeepSeek V4 Flash Vision-Exp MXFP4, `--vision` server mode, `--batched-session 2`, disk KV enabled (`--kv-disk-dir`, 1 TB budget).
## Summary
For multimodal (vision) chat requests, KV reuse works perfectly while the
conversation round-trips exactly (`multimodal live kv hit identity=fingerprint-match`,
sub-second turns on a ~335k-token prompt). But as soon as one request diverges
from the live session by even a handful of tokens at the tail, the server falls
back to a full cold prefill from token 0 — three reuse mechanisms that exist for
text sessions are all unavailable or inert for multimodal ones:
1. **No common-prefix rewind on live token-mismatch.** The live-prefix check
computes the common prefix length but only accepts it on an exact full match:
```
ds4-server: live kv cache miss live=340263 prompt=340551 common=340144 vision=match reason=token-mismatch
ds4-server: chat ctx=0..340551:340551 TOOLS prompt start
```
99.96% of the prompt (340,144 / 340,551 tokens) was identical to the live
session, yet the server restarted prefill from 0. The GLM path right above
it does exactly this kind of rewind (`rewound GLM live prefix from %d to %d`,
`memory-rewind` cache source). The DS4 path is all-or-nothing:
```c
cached = common == old_pos && j->req.prompt.len >= old_pos ? common : 0;
```
2. **Disk KV load is gated off for multimodal requests** (ds4_server.c, request
path around the live-miss fallback):
```c
if (!multimodal && s->kv.enabled && cached == 0 && old_pos >= s->kv.opt.min_tokens) {
kv_cache_store_current(s, slot, "evict");
}
if (!multimodal && cached == 0) {
disk_cached = kv_cache_try_load(...);
}
```
So a vision session that misses the live cache never consults the disk store
at all, even when a matching checkpoint exists.
3. **No `continued` disk checkpoints are written during multimodal cold
prefills.** On `c0a6119` the same workload logged
`kv cache stored tokens=... reason=continued` every ~30k tokens during a long
prefill. On `6289c51`, three full ~330k-token prefills (18 minutes each)
wrote zero checkpoints — so even if (2) were reachable, there would be
nothing fresh to load. (The text-prefix memory path did resume one later
turn at 100,352 of 346,533 tokens, so partial reuse is not entirely dead,
but the deep disk checkpoints are never written.)
## Trigger
A coding-agent client (OpenAI chat-completions, tools, one screenshot per turn)
occasionally re-serializes the previous assistant reply slightly differently
than the server's generated token stream (thinking/DSML round-trip). The
divergence lands inside the previous reply's tail (e.g. 119 tokens before the
end of a 340,263-token history). With text sessions this costs nothing; with a
vision session every such turn is a full ~330k prefill ≈ 18 minutes on an
M3 Ultra.
## Suggested fixes
- On live token-mismatch with `vision=match`, rewind the DS4 session to
`common` (same approach as the existing GLM `memory-rewind` path) instead of
discarding the whole session.
- Allow disk-KV load/store for multimodal sessions (store text-prefix KV even
if image embeddings can't be serialized; the embedding cache already covers
re-encoding cheaply).
## Sanitized log excerpt
```
21:47:24 multimodal live kv hit images=1 cached=334535 prompt=334603 identity=fingerprint-match
21:47:25 chat ctx=334535..334603:68 TOOLS prompt done 1.041s # ~1 s turns while round-trip is exact
...
21:55:10 live kv cache miss live=340263 prompt=340551 common=340144 vision=match reason=token-mismatch
21:55:10 chat ctx=0..340551:340551 TOOLS prompt start # full 18-min prefill, zero continued stores
```
(Happy to provide longer excerpts privately if useful.)
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.