DeepSeek-V4 cannot load on the default configuration: its own factory publishes fp8_ds_mla and ApplyCacheDType refuses it (no store/read, owed to W5)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: KV-DSV4-MULTICACHE
MEASURED on the real 97.68 GiB nvidia/DeepSeek-V4-Flash EXL3 artifact under an
rc lease on thor:gpu0, 2026-09-01, source 693f17e08, through vllm-cli
with NO flags:
[vt load] dsv4-exl3: coalesced TP1 tower resident_bytes=87994957824 (81.952 GiB)
over 43 layers, tp4->tp1, 3-bit trellis; carried host tower
host_bytes=16885558876 (15.726 GiB); host MemAvailable=101.407 GiB
vllm-cli: model load failed (status 2): vllm_engine_load: vt: cache_dtype: an MLA
KV cache has its own quantized page formula upstream (fp8_ds_mla,
kv_cache_interface.py:398-410). W1 landed that page formula but no fp8_ds_mla
store or read, so requesting it here would size the page for bytes nothing
writes; run the MLA model on --kv-cache-dtype auto
Nobody requested it. vllm-cli has no --kv-cache-dtype flag and none was
passed. The refusal fires on the DEFAULT path, and the message misattributes the
cause to an operator request.
The mechanism is a contradiction between two correct-looking pieces:
MakeDeepseekV4KVCachepublishes an fp8_ds_mla topology on purpose,
mirroring upstream, whereuse_fp8_ds_mla_layoutisClassVar[bool] = True
and_resolve_dsv4_kv_cache_dtypewritescache_dtype = "fp8_ds_mla"back
onto the cache config (attention.py:89-119, 140). Its specs carry
vt::DType::kI8.ApplyCacheDType's early-out needsspec.dtype == resolved.storage
(kv_cache_interface.cpp:432-435). Onauto,resolved.storageis the model
dtype (bf16), the spec's iskI8, so the early-out misses and
RetypeAttentionSpecrefuses everyMLAAttentionSpec.
So the model's own factory declares a layout the retype path then refuses, and
the operator has no way to ask for anything different.
The refusal is CORRECT and must not be widened to make this load. W1 landed
the fp8_ds_mla page formula and neither the store nor the read
(kv_cache_interface.cpp:377-388 says so). Accepting it would size every MLA
page at 584 bytes per token while the attention block still writes a bf16 latent
into it -- wrong tokens, not a crash. Making this green by relaxing the guard is
exactly the "never make a red gate green by widening its scope" case.
What is owed is the store and read side, which kv_cache_interface.cpp:388
already names as W5's.
Two things worth separating for whoever takes it:
- The message should stop saying "requesting it here". On this path nothing was
requested, and the next reader will look for a flag they never passed. - The load-blocking question is whether DeepSeek-V4 can serve at all before W5.
If the plain 512B-aligned bf16 MLA arm is servable, publishing it is the
smaller path to a first token; that arm is currently NOT published and is
recorded as owed to W5 indeepseek_v4_registry.cpp:210-211.
Found by the default-configuration load probe for #2441, which this now
supersedes as the load blocker: #2441's block_size refusal is GONE from this
run (0 occurrences), and the load proceeds past the KV geometry to here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with kv_cache_interface.cpp:377-388 and 432-435, then inspect attention.py:89-119,140 and deepseek_v4_registry.cpp:210-211. Reproduce the default vllm-cli load probe and determine whether the plain bf16 MLA arm or W5's store/read path is required. Done means the default load no longer gives a misleading request error and does not accept an unsafe fp8_ds_mla path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100