antirez / antirez/ds4

Server: multimodal sessions lose all KV reuse on a tail token mismatch — no common-prefix rewind, no disk-KV fallback, no continued disk stores

Open
#1,007 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.3k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
4

Description

**Title:** Server: multimodal sessions lose all KV reuse on a tail token mismatch — no common-prefix rewind, no disk-KV fallback, no continued disk stores

**Observed on:** `6289c51` (current main), DeepSeek V4 Flash Vision-Exp MXFP4, `--vision` server mode, `--batched-session 2`, disk KV enabled (`--kv-disk-dir`, 1 TB budget).

## Summary

For multimodal (vision) chat requests, KV reuse works perfectly while the
conversation round-trips exactly (`multimodal live kv hit identity=fingerprint-match`,
sub-second turns on a ~335k-token prompt). But as soon as one request diverges
from the live session by even a handful of tokens at the tail, the server falls
back to a full cold prefill from token 0 — three reuse mechanisms that exist for
text sessions are all unavailable or inert for multimodal ones:

1. **No common-prefix rewind on live token-mismatch.** The live-prefix check
computes the common prefix length but only accepts it on an exact full match:

```
ds4-server: live kv cache miss live=340263 prompt=340551 common=340144 vision=match reason=token-mismatch
ds4-server: chat ctx=0..340551:340551 TOOLS prompt start
```

99.96% of the prompt (340,144 / 340,551 tokens) was identical to the live
session, yet the server restarted prefill from 0. The GLM path right above
it does exactly this kind of rewind (`rewound GLM live prefix from %d to %d`,
`memory-rewind` cache source). The DS4 path is all-or-nothing:

```c
cached = common == old_pos && j->req.prompt.len >= old_pos ? common : 0;
```

2. **Disk KV load is gated off for multimodal requests** (ds4_server.c, request
path around the live-miss fallback):

```c
if (!multimodal && s->kv.enabled && cached == 0 && old_pos >= s->kv.opt.min_tokens) {
kv_cache_store_current(s, slot, "evict");
}
if (!multimodal && cached == 0) {
disk_cached = kv_cache_try_load(...);
}
```

So a vision session that misses the live cache never consults the disk store
at all, even when a matching checkpoint exists.

3. **No `continued` disk checkpoints are written during multimodal cold
prefills.** On `c0a6119` the same workload logged
`kv cache stored tokens=... reason=continued` every ~30k tokens during a long
prefill. On `6289c51`, three full ~330k-token prefills (18 minutes each)
wrote zero checkpoints — so even if (2) were reachable, there would be
nothing fresh to load. (The text-prefix memory path did resume one later
turn at 100,352 of 346,533 tokens, so partial reuse is not entirely dead,
but the deep disk checkpoints are never written.)

## Trigger

A coding-agent client (OpenAI chat-completions, tools, one screenshot per turn)
occasionally re-serializes the previous assistant reply slightly differently
than the server's generated token stream (thinking/DSML round-trip). The
divergence lands inside the previous reply's tail (e.g. 119 tokens before the
end of a 340,263-token history). With text sessions this costs nothing; with a
vision session every such turn is a full ~330k prefill ≈ 18 minutes on an
M3 Ultra.

## Suggested fixes

- On live token-mismatch with `vision=match`, rewind the DS4 session to
`common` (same approach as the existing GLM `memory-rewind` path) instead of
discarding the whole session.
- Allow disk-KV load/store for multimodal sessions (store text-prefix KV even
if image embeddings can't be serialized; the embedding cache already covers
re-encoding cheaply).

## Sanitized log excerpt

```
21:47:24 multimodal live kv hit images=1 cached=334535 prompt=334603 identity=fingerprint-match
21:47:25 chat ctx=334535..334603:68 TOOLS prompt done 1.041s # ~1 s turns while round-trip is exact
...
21:55:10 live kv cache miss live=340263 prompt=340551 common=340144 vision=match reason=token-mismatch
21:55:10 chat ctx=0..340551:340551 TOOLS prompt start # full 18-min prefill, zero continued stores
```

(Happy to provide longer excerpts privately if useful.)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.