fix(KV-MAMBA-ALIGN): prefix caching still defaults OFF for hybrid models, which upstream now defaults ON
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: KV-MAMBA-ALIGN
Found by wave PORTQ-2 re-deriving PORT-NOW entries 62 and 65 of
5559679229..e126687a9a (#2646).
Upstream: f9c74b4b9c vllm#50991, "[Mamba] enable prefix cache by default";
related, and deliberately filed here too: c2881ce603 vllm#50432, "Fix
cross-block race on num_accepted in MRv2 align prefix cache".
1. The exclusion upstream removed is here verbatim
src/vllm/entrypoints/model_loader.cpp:1346-1356:
bool LoadedEngine::ResolveEnablePrefixCaching(const EngineParams& params,
const ModelInfo& model_info) {
if (params.enable_prefix_caching.has_value()) {
return *params.enable_prefix_caching;
}
...
return !model_info.is_hybrid && !model_info.has_inner_state;
}
!model_info.is_hybrid is the C++ spelling of the is_hybrid exclusion this
commit deletes, so hybrid and Mamba models default prefix caching off here and
on upstream at the target.
2. The commit's second half has no resolver here
Upstream also moves MambaModelConfig's documented default cache mode from
"all" to "align". Nothing in this tree ever selects a mode: every production
registry builds MambaSpec with "none" — explicitly at
src/vllm/model_executor/models/qwen3_5_common.cpp:89, and by the constructor
default (include/vllm/v1/kv_cache_interface.h:469) for
kimi_linear_registry.cpp:153, nemotron_h_registry.cpp:317,
glm5_next_registry.cpp:434 and qwen4_exp_registry.cpp:1267.
.agents/engine-matrix.md:101 already records this as item (1) of this row's
debt, in these words: "no config path SELECTS align — MambaSpec defaults
"none" and no caller overrides it, so production is effectively mode-none".
3. Why the one-line flip is not landable alone
Flipping model_loader.cpp:1355 turns prefix caching on for hybrid models while
every MambaSpec still says "none" and while the runner-side align state copy
does not exist — this row's recorded items (1) and (2). The realistic unit is the
row, not the line.
4. c2881ce603 vllm#50432 belongs to this row too, and it is a WARNING
The V2 align postprocess kernel it fixes is surface-absent here — zero hits for
postprocess_mamba, aligned_new_computed, num_accepted_out,
state_idx_gpu — because the runner-side recurrent state copy the align path
presumes does not exist, which is engine-matrix.md:101 item (2) again.
Whoever ports that state copy must port the POST-fix shape: read
num_accepted from a snapshot buffer and always write through the output
pointer. Mirroring the pre-fix form would reintroduce upstream's cross-block race
on the same counter. This note is on this issue rather than only in a sync report
precisely so it is met by the person doing the work.
Not settled
Whether upstream's post-commit default is safe for THIS tree's align allocator
without the runner-side state copy. What a mode-"none" mamba group does under an
enabled prefix cache here was not traced, and that is what would settle the
ordering of §1 against §3.
Nothing was executed for this finding: no build, no test, no GPU.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with item (1) in .agents/engine-matrix.md, then inspect ResolveEnablePrefixCaching in src/vllm/entrypoints/model_loader.cpp and the MambaSpec construction sites listed in the issue. Trace how mode "none" and runner-side state are handled before changing defaults; done requires a settled ordering for hybrid prefix caching, an effective Mamba cache mode, and a race-safe align state copy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100