mudler / mudler/vllm.cpp

MODEL-MM-GLM53-FLASH-CUDA: GLM-5.3-Flash has no device arm, and it is a port onto existing kernels rather than a kernel campaign

Open
#2,410 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: MODEL-MM-GLM53-FLASH-CUDA

Glm5NextForConditionalGeneration (GLM-5.3-Flash) has no device arm.
glm5_next_forward.cpp:231-238 refuses a non-CPU queue by name, and
--device cuda has never produced a token from this model. The only generation
this row has ever observed is O30's Paris. on dgx:gpu0 with --device cpu
at 195.5 s/token.

.agents/specs/glm5-next-flash.md §W9c priced that debt as a kernel campaign on
the premise that the six forward files "carry ZERO vt::Tensor". Measured on
0b4766c96, they carry 25 across 2,783 lines — 15 in glm5_next_moe.cpp
and 10 in glm5_next_kda.cpp, which also make five vt:: compute-op calls
between them. Every primitive family this model needs already has a registered
CUDA provider, and the sibling GlmMoeDsaForCausalLM already drives
mla::ForwardMlaAttentionBlock on a GPU through a seam W3 widened to admit this
row's NoPE geometry. The debt is a PORT, not a kernel campaign.

This issue tracks the port, which the rescoping splits into four waves:

  • W9b — keep-quant device residency of the 101.24 GiB artifact via
    dense_attn::ResidentWeight. Open question: unified memory on GB10.
  • W9c-1 — retire glm5_next_attn.cpp + glm5_next_dsa.cpp's hand-rolled
    MLA/DSA block (991 host lines) onto mla::ForwardMlaAttentionBlock. 11 of 45
    layers. This is also the AGENTS.md §"Shared seams" parallel-path defect (O33).
  • W9c-2 — lift the two CPU-only refusals (glm5_next_kda.cpp:322-325,
    glm5_next_moe.cpp:222-225) so the KDA and MoE arms reach the CUDA providers
    they already call. 34 of 45 layers plus every sparse block.
  • W9c-3 — the compose: Dev/DBuf, a device-carrying ForwardLogits, the
    input.gather_logits dispatch predicate with a residency clause, RMSNorm /
    embedding / lm_head onto their ops, and the mHC decision (O34: mHC has a CUDA
    kernel and no CPU registration). This wave deletes the refusal.

None is started. The rescoping itself lands no product code.

Fleet constraint measured while scoping: on thor:gpu0 (compute_cap 11.0) a
CUDA build with -DVLLM_CPP_FLASH_ATTN=ON and CUTLASS 4.5.0 configures to
CUDA FA2 compiled-arch manifest: [] — FA2's arch table is
8.0,8.6,8.7,8.9,12.0a,12.1a (cmake/CudaArchFeatures.cmake:349). MLA prefill
on this family IS FlashAttention with no fallback beneath it, so the eventual
--device cuda end-to-end test can only ever run on dgx:gpu0 (sm_121a).

Owed entries: O32 (the arm, as four waves), O33 (the parallel path), O34 (mHC's
inverted gap), O35 (two drifted anchors in this row's own briefing).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with glm5_next_forward.cpp:231-238 and the refusal sites in glm5_next_kda.cpp:322-325 and glm5_next_moe.cpp:222-225. Compare the sibling GlmMoeDsaForCausalLM path and the four W9b/W9c waves, including mla::ForwardMlaAttentionBlock and the device-carrying compose. Done means GLM-5.3-Flash generates a token with --device cuda on dgx:gpu0, with the listed device-residency and dispatch paths enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.