No production path gives an EXL3 checkpoint a paged KV cache: DeepseekV4ForwardExl3Paged has only test callers
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: MODEL-DSV4-EXL3
DeepseekV4ForwardExl3Paged is the only DeepSeek-V4 forward that takes both a
paged KV cache and a DeepseekV4CompressorState. It has NO production caller.
$ grep -rn "DeepseekV4ForwardExl3Paged" src/ examples/
src/vllm/model_executor/models/deepseek_v4.cpp:3738 (its own refusal message)
src/vllm/model_executor/models/deepseek_v4.cpp:3740 (its own refusal message)
Every other reference is in tests/vllm/models/test_deepseek_v4_exl3_loader.cpp.
The registered forward ForwardDeepseekV4ForCausalLM
(deepseek_v4_registry.cpp:102-133) has three branches, and none of them reaches
it:
input.gather_logits->DeepseekV4Model::ForwardDevice, which binds
dev_be.exl3and neitherpaged_kvnorcompressor.input.multi_kv != nullptr->DeepseekV4ForwardGgufPaged, which refuses by
name on!weights.has_gguf_weights. An EXL3 checkpoint has none, so this
branch is a refusal for this arm rather than a path.- otherwise ->
DeepseekV4Model::Forward->DeepseekV4ForwardExl3, which also
binds neither.
So no production path gives an EXL3 checkpoint a paged KV cache. V4Backend
documents what that means: "Null = stateless full-recompute (the default / --gpu
path)". The EXL3 arm therefore recomputes the entire prefix on every decode step.
WHY THIS MATTERS MORE THAN #2442. The published 44-47 tok/s target is measured at
384k context, 1 seq. A forward that recomputes 384k tokens per step is not a
decode, and no amount of expert-tower residency or speculation changes that.
#2442 (routed experts on a CPU queue) is real and now fixed, but this sits in
front of it.
It also means the DSA composition -- the compressor and indexer work that W3
landed -- is unreachable in production, since compressor is only ever non-null
on the paged entry. Its gates are real; its only callers are tests.
AGENTS.md §"Nothing lands dead": a staged slice may land unreached only when
the commit body, the PR body and the row's ## Owed all name it. This one is not
recorded anywhere as unreached.
Found while wiring #2442, by checking whether the device-staging call sites were
on the hot path.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ForwardDeepseekV4ForCausalLM in src/vllm/model_executor/models/deepseek_v4_registry.cpp and trace its three branches into the DeepSeek-V4 forward functions in deepseek_v4.cpp. Compare the production paths with DeepseekV4ForwardExl3Paged and its callers in tests/vllm/models/test_deepseek_v4_exl3_loader.cpp. Done means an EXL3 checkpoint has a production path that supplies paged KV and compressor state, with tests covering that path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100