antirez / antirez/ds4

Disk KV cache can be reused across different model weights with the same model_id

Aperta
#805 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C
Stelle
22.3k
Fork
2.1k
Merge medio
1g 3h
PR unite (30g)
4

Descrizione

I found a potentially unsafe disk KV cache reuse case when switching between different GGUF weights that share the same DS4 model structure.

### Setup

- macOS / Metal
- DeepSeek V4 Flash 0731 official GGUF
- Huihui DeepSeek V4 Flash 0731 variant
- Both launched with the same `--kv-disk-dir`
- Same context / prompt format / quantization-compatible cache settings

### Observed behavior

After running the official model and creating an 81,920-token disk checkpoint, I stopped the server, switched `ds4flash.gguf` to the Huihui model, and restarted DS4 using the same KV cache directory.

The Huihui model immediately accepted the checkpoint created by the official model:

```text
kv cache hit text tokens=81920 ... quant=2 ...
```

So a checkpoint produced using one set of model weights can apparently be restored while another set of weights is loaded, as long as the model structure / model_id and other cache compatibility checks match.

### Concern

KV states depend on the actual model weights. Two GGUFs can have the same architecture, tokenizer and DS4 `model_id` while containing different weights, such as fine-tuned or abliterated variants.

In that case the cache can be structurally compatible while semantically invalid, potentially causing silent output corruption rather than an explicit error.

Cross-quant reuse for different quantizations of the same underlying model makes sense, but it seems useful to distinguish that from reuse across genuinely different model weights.

### Possible fix

Could disk KV metadata include a stable model-weight fingerprint, or another model identity derived from the GGUF, and require it to match on restore?

Ideally the identity would remain compatible across supported quantizations of the same underlying weights if cross-quant reuse is intentional.

### Workaround

For now I am using separate cache directories:

```text
/tmp/ds4-kv-official
/tmp/ds4-kv-huihui
```

I can provide additional trace output or test a proposed fix if useful.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.