VT_KV_CACHE_F32=1 cannot complete a forward: reshape_and_cache refuses f32 cache against bf16 k/v
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: -
Owned under ## Owed in
.agents/specs/qwen38-27b-q4km-token-exactness.md.
VT_KV_CACHE_F32=1 is a documented same-binary A/B lever
(include/vllm/v1/kv_cache_dtype.h:28, "VT_KV_CACHE_F32=1 selects f32 for the
A/B"). On the Qwen3.8-27B Q4_K_M GGUF CPU path it cannot complete a single
forward.
Measured 2026-09-02, thor:gpu0, rc job
25cbca74-df63-43e4-b7db-1315497c4d41, worker rc-worker-n8smh, on
vllm-bench --model Qwen3.8-27B-Q4_K_M.gguf (CPU-only Release build):
engine-fatal: EngineCore busy loop threw: vt: reshape_and_cache:
k/v/k_cache/v_cache must share one float dtype (auto cache path)
at src/vt/ops.cpp:3947
RUNMEM child_rc=1 wall_s=36.923
ResolveKvCacheDType turns the cache pages f32, but the attention path still
produces bf16 k and v, so reshape_and_cache's "one float dtype" guard
refuses the pair. The lever changes the store dtype without changing the
producer.
The focused unit suite does NOT catch this. test_qwen27_paged_forward is
33/33 and 780/780 green under VT_KV_CACHE_F32=1; only the real engine path
dies. So the knob reads as gated and is not.
This predates the work that found it — nothing in
row/QUANT-Q4K-TOKEN-EXACT touches the KV cache dtype. It is filed rather than
fixed in-flow because the fix is not a dtype change at the call site: it needs
the attention path to emit k/v in the resolved cache dtype (or
reshape_and_cache to accept a widening store), which is its own change with its
own gate.
Consequence for #2534. The KV-cache term of that arm's bf16-vs-f32 question
cannot be isolated until this runs. llama.cpp's CPU KV cache defaults to
GGML_TYPE_F16 (src/llama-context.cpp:3538-3539) against our bf16, so this is
a real precision difference against that arm's oracle and it is currently
unmeasurable from our side.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with the Qwen3.8-27B-Q4_K_M CPU benchmark under VT_KV_CACHE_F32=1, then inspect include/vllm/v1/kv_cache_dtype.h and src/vt/ops.cpp around reshape_and_cache. Compare the real engine path with test_qwen27_paged_forward; done means a single forward completes with the lever enabled and the focused suite remains green.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100