mudler / mudler/vllm.cpp

MODEL-MM-GLM53-FLASH-CUDA: GLM-5.3-Flash's k-pool indexer has no device op, so the device arm is not a routing-only port

Open
#2,415 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: MODEL-MM-GLM53-FLASH-CUDA

.agents/specs/glm5-next-flash.md §W9c's rescoping (#2410, #2413) priced the
device arm as "a PORT, not a kernel campaign", on the finding that "Every
primitive family this model needs already has a registered CUDA provider" and
that therefore no kernel needs writing for correctness.

That is false for one family, and it is the family on the critical path of all
11 DSA layers.

GLM-5.3-Flash's indexer is a k-pool indexer, and nothing in this tree implements it on a device

Measured on e4aa7c527.

  • include/vt/ops.h:130-131 defines exactly two indexer ops, kDsaIndexerLogits
    and kDsaTopkSelect. Its own comment at :123 names them "the DSA 'Lightning
    Indexer' selection pair". kDeepseekV4Dsa (:295) is the same family
    (:284-285). There is no pooled-selection op of any kind.
  • git grep -l 'kpool\|index_kpool\|compress_ape\|compress_gate' src/ include/
    returns 11 files, and every one is a glm5_next_* file. Zero in the shared
    seam, zero under src/vt/, zero in any other model.
  • mla::MlaBlockWeights carries exactly five indexer tensors
    (mla_attention.h:510-514). This model has all five plus kpool_ape
    [index_kpool, head_dim] and kpool_gate [head_dim, hidden_size]
    (glm5_next_dsa.h:137, :141) — the two learned weights that ARE the
    pooling. They are named tensors in the published artifact,
    indexer_compressor_ape.weight and indexer_compressor_gate.weight, loaded
    at glm5_next_loader.cpp:378-380.

The selection is structurally different rather than a variant. The seam picks
the top index_topk tokens from per-token logits. This model pools
index_kpool = 4 consecutive valid tokens under a learned per-channel 4-way
softmax (glm5_next_dsa.cpp:168-304), picks the top
index_topk / index_kpool = 512 pools, expands them back to member token
indices, then appends the ragged visible tail raw and unscored (:307-362). The
output is 2051 wide, not 2048 (OutputWidth(), glm5_next_dsa.cpp:56-59),
and it carries -1 sentinels and duplicates, which upstream absorbs with
scatter_add_ + ne(0).

glm5_next_dsa.cpp carries zero vt::Tensor across its 534 lines.

What this costs

The 11 DSA layers cannot reach a GPU by routing alone. Either

  1. a k-pool indexer CUDA kernel is written (a new op family: the pooled
    compression, the pool-level top-k, and the visible-tail append), or
  2. the indexer stays on the host, and every DSA layer costs a device-to-host
    round trip on every step — 11 per step, on the critical path.

Neither is priced in .agents/specs/glm5-next-flash.md today. This issue owns
the measurement and the decision.

What it also corrects

O33 states that glm5_next_attn.cpp + glm5_next_dsa.cpp are 991 host lines
reimplementing a shipped block. It is 457 lines, not 991. glm5_next_dsa.cpp
duplicates nothing, because the seam does not implement what it computes. The
parallel-path defect is glm5_next_attn.cpp alone. O33's text lands with
#2413; the correction is recorded in the row's spec under
"### W9c-1 — the MLA route, PRICED and REFUSED".

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .agents/specs/glm5-next-flash.md §W9c and inspect include/vt/ops.h, glm5_next_dsa.cpp, glm5_next_dsa.h, and glm5_next_loader.cpp. Compare the host k-pool selection path with registered device operations, then measure the cost of the host round trips versus the missing device operation. Done means the spec records a supported decision and its impact on all 11 DSA layers.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.