mudler / mudler/vllm.cpp

routed_scaling_factor is applied to the routing weights, not the routed output: the deviation is live at GLM's 1.8

Open
#2,930 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: - (owned under ## Owed in .agents/specs/glm4-moe-lite-router-f32.md)

src/vllm/model_executor/models/deepseek_v2.cpp:372 applies
routed_scaling_factor to the routing weights:

args.routed_scaling_factor = p.routed_scaling_factor;  // DEVIATION (a)

Upstream applies it to the routed output on this path. Glm4MoE passes
apply_routed_scale_to_output=not self.is_rocm_aiter_moe_enabled
(vllm/model_executor/models/glm4_moe.py:201, true on CUDA), and
DeepseekV2MoE passes the same at vllm/model_executor/models/deepseek_v2.py:1266.
vllm/model_executor/layers/fused_moe/layer.py:292-300 then hands the ROUTER
routed_scaling_factor=1.0 — a nop inside grouped_topk
(vllm/model_executor/layers/fused_moe/router/grouped_topk_router.py:157-159) —
and :411-417 hands the scale to the runner instead, where
vllm/model_executor/layers/fused_moe/runner/moe_runner.py:390-400,715 does
fused_output *= routed_scaling_factor on the combined routed output before the
shared expert is added.

The two are algebraically identical and differ only in where the rounding falls:
upstream rounds the weighted sum to the output dtype and then scales, we scale
the f32 weights and round once.

deepseek_v2.cpp:42-50 records the deviation and states why it was safe to
carry: DeepSeek-V2-Lite has routed_scaling_factor: 1.0, so on the row's own
gate vehicle the two forms are bit-identical. zai-org/GLM-4.7-Flash has
routed_scaling_factor: 1.8
, so on Glm4MoeLiteForCausalLM the deviation is
live and has never been exercised against the oracle by anything that could see
it. Every other noaux_tc checkpoint with a scale other than 1.0 is in the same
position.

The seam already exists, which narrows what the repair is. vt::MoeCombine
already takes a trailing routed_scale documented as "the routed_scaling_factor
applied to the ROUTED sum before the shared term is added"
(include/vt/ops.h:2449-2452), so this is not a missing capability: it is a
choice of which of two existing knobs deepseek_v2.cpp sets.

Not repaired in the flow that found it (#2928) all the same. Moving the scale
changes the numerical output of every model that routes through this block —
DeepSeek-V2 among them, where it is currently a no-op only because that
checkpoint's factor is 1.0 — so it needs its own spec, a red-before measurement
of the two forms at a factor other than 1.0, and a fresh review, rather than
riding a one-field dtype repair.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at src/vllm/model_executor/models/deepseek_v2.cpp:372 and compare the routing and output-scale paths in vllm/model_executor/models/glm4_moe.py:201, deepseek_v2.py:1266, and fused_moe/layer.py:292-300,411-417. Read include/vt/ops.h:2449-2452 and moe_runner.py:390-400,715, then measure both forms at a factor other than 1.0. Done means the spec records the chosen behavior and the red-before measurement supports the reviewed repair.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.