mudler / mudler/vllm.cpp

Four models bypass the typed HfConfig via cfg.raw, so normalization does not reach them

Open
#2,963 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: ENG-ATTENTION-WINDOW

What

Four models read a config value by falling back to the raw checkpoint
document
, which defeats every normalisation LoadHfConfig performs on the
typed field:

g.sliding_window = cfg.sliding_window.value_or(RawInt(cfg.raw, "sliding_window", 0));
  • src/vllm/model_executor/models/gemma2.cpp:103
  • src/vllm/model_executor/models/gemma3.cpp:101
  • src/vllm/model_executor/models/gemma4.cpp:139
  • src/vllm/model_executor/models/laguna_weights.cpp:117

cfg.raw is the full untouched JSON (hf_config.cpp:598, nlohmann::json raw; // full doc for fields we don't type yet). So when the typed field is
std::nullopt, these four do not get "absent" — they get whatever the checkpoint
literally said.

Why it matters, with the case that found it

HfConfig normalises a checkpoint sliding_window == 0 to std::nullopt
(hf_config.cpp:80-87), mirroring ModelConfig's own conversion. At these four
sites that normalisation is undone: value_or falls through to
RawInt(raw, "sliding_window", 0) and reads the 0 straight back.

That specific case is currently benign by luck — the call sites guard
*sliding_window > 0, so a 0 is inert either way.

It is not benign for the mechanism upstream actually uses to disable the
window.
vllm/config/model.py:766-769 implements
ModelConfig.disable_sliding_window by setting hf_text_config.sliding_window = None. Transcribed here, that would null the typed field and these four models
would keep their window from cfg.raw, silently. ENG-ATTENTION-WINDOW W3 hit
this and had to consume its switch at the point of USE
(ResolveAttentionWindow) instead of at the point of configuration.

So the bug is not "a wrong window today". It is that the typed config is not
authoritative
, and any future normalisation, override, or CLI flag that works
by writing the typed field will appear to work and will not.

Scope

sliding_window is the instance that was found. The value_or(RawInt(raw, ...))
shape should be swept for — it is a general pattern, not a one-off, and each
instance is a place a normalised field can be bypassed.

Not fixed in flow

Repointing four models off cfg.raw changes what each reads when a checkpoint
and the typed parse disagree, which is a behaviour change per model with its own
gate. It is filed rather than folded into W3, whose switch is correct without it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the four call sites in src/vllm/model_executor/models/gemma2.cpp, gemma3.cpp, gemma4.cpp, and laguna_weights.cpp, then inspect hf_config.cpp:80-87 and :598 to understand typed normalization and cfg.raw. Sweep for the same value_or(RawInt(raw, ...)) pattern; done means every relevant normalized field remains authoritative without folding this work into W3.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.