mudler / mudler/vllm.cpp

dots3-note vision MoE: the SOFTMAX router and the topk<2 arm are refused, because the self-normalizing combine needs a per-token divisor

Open
#2,615 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: MODEL-MM-dots3-note-dots3-note-for-causal-lm

W6b (#2613) lifts the pyramid MoE ViT refusal and implements the released
vision_config's router arm exactly: router_scoring_func = "sigmoid",
capacity_factor = 2 and so topk = min(int(2), num_routed) = 2 on every one of
the 17 routed blocks. Two upstream router arms are still refused BY NAME, and
this issue owns them.

What is refused

Dots3NoteVisionRefusal (src/vllm/model_executor/models/dots3_note_vision.cpp)
returns a message naming this issue when either holds:

  • router_scoring_func is not "sigmoid" — upstream's other branch is
    torch.softmax(gate_logits, dim=-1, dtype=torch.float32) (vision.py:185-186
    @ 9035151d6).
  • any routed block's topk = min(int(capacity_factor), num_routed) is below 2.

Why they are not in W6b

Upstream's combine is SELF-NORMALIZING: it accumulates a per-token
aggregated_gate and divides the routed sum by aggregated_gate + 1e-9
(vision.py:207-218 @ 9035151d6). On the sigmoid arm with topk > 1 upstream
has already renormalized the routing weights to sum to router_scale
(vision.py:182-184), so that denominator is a per-tower CONSTANT and W6b folds
it into vt::MoeCombine's routed_scale — an exact fold, not an approximation.

On the other two arms the denominator is a PER-TOKEN value, and no shared op in
include/vt/ops.h expresses a per-token scale on the combine: MoeCombine's
routed_scale is one float and MulColVecF32 broadcasts over COLUMNS, not
rows. Serving them would mean either extending vt::MoeCombine with a per-token
scale tensor on both the CPU and CUDA kernels — an edit to the op DeepSeek-V2's
SACRED token-exact path routes through — or a host round-trip per routed block
to renormalize a [L, top_k] f32 buffer, which is a device sync per block on
the image path.

Neither is W6b's scope (#2613 fixes the arm at sigmoid / capacity 2), and no
published dots3-note checkpoint selects either: dots-studio/dots3-note-prev
sets router_scoring_func = "sigmoid" and capacity_factor = 2, and
DotsMoEVitConfig's own defaults are the same (vision.py:51-52).

Closing it

Extend vt::MoeCombine with an optional [T] f32 per-token divisor (CPU and
CUDA kernels, red-first), route both arms through it, and delete the refusal in
the same change so message and behaviour cannot drift. The gate is the same
consistency gate W6b uses — the in-test double-precision reference already
spells upstream's literal aggregated_gate division, so it measures the new arm
with no change.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/vllm/model_executor/models/dots3_note_vision.cpp, include/vt/ops.h, and the existing W6b consistency gate. Trace vt::MoeCombine through its CPU and CUDA kernels, then compare the test's double-precision reference with upstream vision.py:207-218. Done means both refused router arms use the per-token divisor, the CPU and CUDA paths agree, and the refusal is removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.