mudler / mudler/vllm.cpp

defer(MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm): the fused-MoE scale orientation fix is inert behind the BF16-only refusal

Open
#2,667 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: MODEL-TEXT-qwen3-5-qwen3-5-moe-for-causal-lm

Found by wave PORTQ-2 re-deriving PORT-NOW entry 46 of 5559679229..e126687a9a
(#2646). Upstream:
94e9ef0768 vllm#50137, "Don't transpose fused MoE quantization scales in
RoutedExperts.load_weights".

Deferred work with a named gate, filed so the person who lands the gate meets
it.

The gate: quantized stacked experts are refused by dtype, upstream of the

orientation test

src/vllm/model_executor/models/qwen3_5_weights.cpp:965-977 refuses any non-BF16
stacked expert tensor by name ("A quantized stacked expert layout is OWED: see
.agents/specs/moe-bf16-stacked-experts.md"), and that check runs before the
orientation branch. The bug upstream fixes — a fused quantization SCALE of shape
(2*intermediate, 1) being transposed because it carries no hidden dim — needs a
quantized fused tensor to reach the branch, and none can.

The commit's second half is unreachable for a different reason:
hidden_dim_unpadded and unpadded_hidden appear here only inside two quoted
upstream comment lines (qwen3_5_weights.cpp:890,895). There is no
padded/unpadded hidden distinction, so the ... or self.moe_config.hidden_dim
fallback has no null case to cover.

The warning for whoever lands the quantized stacked arm

This tree mirrors the PRE-commit orientation rule and cites upstream's exact
pre-commit lines in its own comments (qwen3_5_weights.cpp:890,895), implementing
them at :986-1005 with "last dim is hidden is checked FIRST".

Port the post-commit rule, not the in-tree one: transpose only when the
hidden dim is definitively on the wrong axis, and fall back to hidden_dim when
the unpadded value is unset. Mirroring what is here today would carry upstream's
bug into the new arm.

One property this tree already has and should keep: a tensor with hidden on
neither trailing dim is REFUSED BY NAME (:989-992) rather than silently
transposed.

Nothing was executed for this finding: no build, no test, no GPU.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read src/vllm/model_executor/models/qwen3_5_weights.cpp:965-1005 and .agents/specs/moe-bf16-stacked-experts.md first, then compare the cited upstream change and its post-commit orientation rule. Done means the quantized stacked-expert arm admits valid tensors, uses the definitive hidden-dimension fallback, and refuses tensors with hidden on neither trailing dimension.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.