mudler / mudler/vllm.cpp

fix(MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm): stacked expert orientation is resolved by sniffing a shape, the heuristic upstream deleted

Open
#2,689 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm

Found by wave PORTQ-4 (#2680)
re-deriving PORT-NOW entry 144 of 5559679229..e126687a9a, upstream
36f4630d89 vllm#50727
"[Bugfix][MoE] Fix fused block-scale orientation".

What is wrong

The stacked-expert loader resolves weight orientation by sniffing a runtime
shape
— the exact heuristic upstream deleted, present here under different names:

src/vllm/model_executor/models/qwen3_5_weights.cpp:988
  const bool gu_canonical = gu.shape[2] == hidden;  // [E, 2I, H]
src/vllm/model_executor/models/qwen3_5_weights.cpp:999
  const bool dn_canonical = dn.shape[1] == hidden;  // [E, H, I]

The comment at :985-987 names it as a transcription of upstream's test and
tie-break, including the degenerate 2I == H case. Upstream replaced this with an
explicit is_fused_checkpoint_transposed flag threaded from the model, precisely
because a shape probe cannot distinguish a transposed checkpoint from a
coincidentally square one.

What is not portable, stated per half

The commit's scale-layout refinement ("scale" not in weight_name or quant_method == BLOCK) is inert here: this loader refuses any non-BF16 stacked expert tensor by
name at qwen3_5_weights.cpp:965-974 ("only BF16 3-D stacked routed experts are
implemented ... A quantized stacked expert layout is OWED"). A stacked checkpoint
never carries a block scale into the orientation code. That half becomes real the
day the quantized stacked layout lands.

Blast radius, stated honestly

This is a latent-ambiguity fix, not a live wrong-logits bug.
.agents/model-matrix.md:321 records both published Qwen indices
(Qwen3.6-35B-A3B, Qwen3.8-2.4T-A95B) as canonical, so no shipped checkpoint
takes the transposed branch today. And the transposed population upstream needed the
flag for is qwen3_vl_moe, which has no MoE port here — qwen3_vl.cpp:122-137
builds a dense text tower.

It is filed because a shape probe that happens to be right on today's two artifacts
is not a guarantee, and because the next stacked checkpoint is the one that finds
out.

Size

Replace the two shape probes with an explicit per-architecture declaration threaded
from the registry, and refuse rather than guess when the declaration and the
shape disagree. ~40-80 lines across qwen3_5_weights.cpp, qwen3_5_weights.h and
the registry that constructs the layout, plus a red-first test.

Verification state

Nothing was executed — no build, no test run, no GPU, no lease. Both cited lines
were printed from the tree at a700e8da6 and re-read by the wave operator.
routed_experts.py is present at 5559679229, so the heuristic is pre-pin
which is why this tree carries it. No prior triage: git log --all --grep 36f4630d89
is empty, and .agents/model-matrix.md:321 documents the stacked arm's landing
(#740) without mentioning the orientation ambiguity.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/vllm/model_executor/models/qwen3_5_weights.cpp:965-999 and qwen3_5_weights.h, then trace the registry that constructs the layout. Add a per-architecture orientation declaration, reject mismatches instead of inferring from shapes, and create the red-first test described by the issue. Verify with the relevant build and test suite.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.