mudler / mudler/vllm.cpp

fix(KV-MAMBA-ALIGN): prefix caching still defaults OFF for hybrid models, which upstream now defaults ON

Open
#2,660 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: KV-MAMBA-ALIGN

Found by wave PORTQ-2 re-deriving PORT-NOW entries 62 and 65 of
5559679229..e126687a9a (#2646).
Upstream: f9c74b4b9c vllm#50991, "[Mamba] enable prefix cache by default";
related, and deliberately filed here too: c2881ce603 vllm#50432, "Fix
cross-block race on num_accepted in MRv2 align prefix cache".

1. The exclusion upstream removed is here verbatim

src/vllm/entrypoints/model_loader.cpp:1346-1356:

bool LoadedEngine::ResolveEnablePrefixCaching(const EngineParams& params,
                                               const ModelInfo& model_info) {
  if (params.enable_prefix_caching.has_value()) {
    return *params.enable_prefix_caching;
  }
  ...
  return !model_info.is_hybrid && !model_info.has_inner_state;
}

!model_info.is_hybrid is the C++ spelling of the is_hybrid exclusion this
commit deletes, so hybrid and Mamba models default prefix caching off here and
on upstream at the target.

2. The commit's second half has no resolver here

Upstream also moves MambaModelConfig's documented default cache mode from
"all" to "align". Nothing in this tree ever selects a mode: every production
registry builds MambaSpec with "none" — explicitly at
src/vllm/model_executor/models/qwen3_5_common.cpp:89, and by the constructor
default (include/vllm/v1/kv_cache_interface.h:469) for
kimi_linear_registry.cpp:153, nemotron_h_registry.cpp:317,
glm5_next_registry.cpp:434 and qwen4_exp_registry.cpp:1267.

.agents/engine-matrix.md:101 already records this as item (1) of this row's
debt, in these words: "no config path SELECTS align — MambaSpec defaults
"none" and no caller overrides it, so production is effectively mode-none".

3. Why the one-line flip is not landable alone

Flipping model_loader.cpp:1355 turns prefix caching on for hybrid models while
every MambaSpec still says "none" and while the runner-side align state copy
does not exist — this row's recorded items (1) and (2). The realistic unit is the
row, not the line.

4. c2881ce603 vllm#50432 belongs to this row too, and it is a WARNING

The V2 align postprocess kernel it fixes is surface-absent here — zero hits for
postprocess_mamba, aligned_new_computed, num_accepted_out,
state_idx_gpu — because the runner-side recurrent state copy the align path
presumes does not exist, which is engine-matrix.md:101 item (2) again.

Whoever ports that state copy must port the POST-fix shape: read
num_accepted from a snapshot buffer and always write through the output
pointer. Mirroring the pre-fix form would reintroduce upstream's cross-block race
on the same counter. This note is on this issue rather than only in a sync report
precisely so it is met by the person doing the work.

Not settled

Whether upstream's post-commit default is safe for THIS tree's align allocator
without the runner-side state copy. What a mode-"none" mamba group does under an
enabled prefix cache here was not traced, and that is what would settle the
ordering of §1 against §3.

Nothing was executed for this finding: no build, no test, no GPU.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with item (1) in .agents/engine-matrix.md, then inspect ResolveEnablePrefixCaching in src/vllm/entrypoints/model_loader.cpp and the MambaSpec construction sites listed in the issue. Trace how mode "none" and runner-side state are handled before changing defaults; done requires a settled ordering for hybrid prefix caching, an effective Mamba cache mode, and a race-safe align state copy.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.