antirez / antirez/ds4

Extend live-prefix rewind (#668) beyond GLM: DeepSeek V4 Flash on Metal still cold-prefills exact-prefix retries

Abierto
#698 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
22.3k
Forks
2.1k
Merge medio
1 d 3 h
PR fusionados (30 d)
4

Descripción

## Summary

#668 added live-KV rewind for repeated exact-prefix prompts (commit 7694112), but `live_prefix_rewind_target` is gated on `ds4_engine_is_glm_dsa()`, so DeepSeek V4 Flash on the Metal backend still does a full cold prefill when a client resends a prompt that is an exact token-level prefix of the live session.

## Reproduction / evidence

Agent client (Hermes Agent, OpenAI chat/completions) aborts a stream mid-generation (user redirect). The slot's live KV then holds `prompt + partial generation`. The resent prompt is an exact prefix of that state — and is treated as a total miss:

```
ds4-server: live kv cache miss live=175286 prompt=175249 common=175249 reason=token-mismatch
ds4-server: chat ctx=0..175249:175249 prompt start
...
ds4-server: chat ctx=0..175249:175249 prompt done 643.560s
```

`common == prompt_len` (the whole 175,249-token prompt matches the live prefix; only the 37 partially-generated tokens diverge), yet the slot is discarded and ~10.7 minutes of prefill are redone. This happened 4 times in 15 minutes in a real session.

## Why this seems generalizable

`ds4_session_rewind()` itself looks engine-agnostic (truncates the checkpoint, invalidates MTP/DSpark capture). The GLM gate appears to guard the dense-cache capping (`ds4_session_glm_cap_dense_cache`). Question: what needs validation to enable the rewind for the Flash/Metal session path — is there a Metal-graph or streamed-expert invariant that a truncated checkpoint violates?

## Environment

- macOS, Apple Silicon
- `ds4-server -m ds4flash.gguf -c 262144 --host 127.0.0.1 --port 8001 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192 --batched-session 4 --mtp .../DeepSeek-V4-Flash-DSpark-support.gguf --dspark`
- Model: DeepSeek V4 Flash
- Client: Hermes Agent (also reproduced the pattern described in #609)

Related: #609 / PR #611 (OpenAI tool-round continuation), #444 (disk KV eviction). Happy to test any branch — the repro environment is live here.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.