routed_scaling_factor is applied to the routing weights, not the routed output: the deviation is live at GLM's 1.8
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: - (owned under ## Owed in .agents/specs/glm4-moe-lite-router-f32.md)
src/vllm/model_executor/models/deepseek_v2.cpp:372 applies
routed_scaling_factor to the routing weights:
args.routed_scaling_factor = p.routed_scaling_factor; // DEVIATION (a)
Upstream applies it to the routed output on this path. Glm4MoE passes
apply_routed_scale_to_output=not self.is_rocm_aiter_moe_enabled
(vllm/model_executor/models/glm4_moe.py:201, true on CUDA), and
DeepseekV2MoE passes the same at vllm/model_executor/models/deepseek_v2.py:1266.
vllm/model_executor/layers/fused_moe/layer.py:292-300 then hands the ROUTER
routed_scaling_factor=1.0 — a nop inside grouped_topk
(vllm/model_executor/layers/fused_moe/router/grouped_topk_router.py:157-159) —
and :411-417 hands the scale to the runner instead, where
vllm/model_executor/layers/fused_moe/runner/moe_runner.py:390-400,715 does
fused_output *= routed_scaling_factor on the combined routed output before the
shared expert is added.
The two are algebraically identical and differ only in where the rounding falls:
upstream rounds the weighted sum to the output dtype and then scales, we scale
the f32 weights and round once.
deepseek_v2.cpp:42-50 records the deviation and states why it was safe to
carry: DeepSeek-V2-Lite has routed_scaling_factor: 1.0, so on the row's own
gate vehicle the two forms are bit-identical. zai-org/GLM-4.7-Flash has
routed_scaling_factor: 1.8, so on Glm4MoeLiteForCausalLM the deviation is
live and has never been exercised against the oracle by anything that could see
it. Every other noaux_tc checkpoint with a scale other than 1.0 is in the same
position.
The seam already exists, which narrows what the repair is. vt::MoeCombine
already takes a trailing routed_scale documented as "the routed_scaling_factor
applied to the ROUTED sum before the shared term is added"
(include/vt/ops.h:2449-2452), so this is not a missing capability: it is a
choice of which of two existing knobs deepseek_v2.cpp sets.
Not repaired in the flow that found it (#2928) all the same. Moving the scale
changes the numerical output of every model that routes through this block —
DeepSeek-V2 among them, where it is currently a no-op only because that
checkpoint's factor is 1.0 — so it needs its own spec, a red-before measurement
of the two forms at a factor other than 1.0, and a fresh review, rather than
riding a one-field dtype repair.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at src/vllm/model_executor/models/deepseek_v2.cpp:372 and compare the routing and output-scale paths in vllm/model_executor/models/glm4_moe.py:201, deepseek_v2.py:1266, and fused_moe/layer.py:292-300,411-417. Read include/vt/ops.h:2449-2452 and moe_runner.py:390-400,715, then measure both forms at a factor other than 1.0. Done means the spec records the chosen behavior and the red-before measurement supports the reviewed repair.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100