mudler / mudler/vllm.cpp

VT_KV_CACHE_F32=1 cannot complete a forward: reshape_and_cache refuses f32 cache against bf16 k/v

Open
#2,548 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: -

Owned under ## Owed in
.agents/specs/qwen38-27b-q4km-token-exactness.md.

VT_KV_CACHE_F32=1 is a documented same-binary A/B lever
(include/vllm/v1/kv_cache_dtype.h:28, "VT_KV_CACHE_F32=1 selects f32 for the
A/B"). On the Qwen3.8-27B Q4_K_M GGUF CPU path it cannot complete a single
forward.

Measured 2026-09-02, thor:gpu0, rc job
25cbca74-df63-43e4-b7db-1315497c4d41, worker rc-worker-n8smh, on
vllm-bench --model Qwen3.8-27B-Q4_K_M.gguf (CPU-only Release build):

engine-fatal: EngineCore busy loop threw: vt: reshape_and_cache:
  k/v/k_cache/v_cache must share one float dtype (auto cache path)
  at src/vt/ops.cpp:3947
RUNMEM child_rc=1 wall_s=36.923

ResolveKvCacheDType turns the cache pages f32, but the attention path still
produces bf16 k and v, so reshape_and_cache's "one float dtype" guard
refuses the pair. The lever changes the store dtype without changing the
producer.

The focused unit suite does NOT catch this. test_qwen27_paged_forward is
33/33 and 780/780 green under VT_KV_CACHE_F32=1; only the real engine path
dies. So the knob reads as gated and is not.

This predates the work that found it — nothing in
row/QUANT-Q4K-TOKEN-EXACT touches the KV cache dtype. It is filed rather than
fixed in-flow because the fix is not a dtype change at the call site: it needs
the attention path to emit k/v in the resolved cache dtype (or
reshape_and_cache to accept a widening store), which is its own change with its
own gate.

Consequence for #2534. The KV-cache term of that arm's bf16-vs-f32 question
cannot be isolated until this runs. llama.cpp's CPU KV cache defaults to
GGML_TYPE_F16 (src/llama-context.cpp:3538-3539) against our bf16, so this is
a real precision difference against that arm's oracle and it is currently
unmeasurable from our side.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with the Qwen3.8-27B-Q4_K_M CPU benchmark under VT_KV_CACHE_F32=1, then inspect include/vllm/v1/kv_cache_dtype.h and src/vt/ops.cpp around reshape_and_cache. Compare the real engine path with test_qwen27_paged_forward; done means a single forward completes with the lever enabled and the focused suite remains green.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.