GLM-5.3 non-flash on gfx1151: the forward dumps core, so no token has been observed since #2511
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-ROCM
Spec: .agents/specs/rocm-mla-attention-nonflash.md
What is settled
#2926 landed native kROCM kMlaPrefillAttention and kMlaDecodeAttention
(src/vt/rocm/rocm_mla_attn.hip, merged at 967c1906f). They are registered,
gated against the CPU oracle, and REACHED on the real 201.83 GiB
unsloth/GLM-5.3-GGUF UD-IQ1_S artifact on strix:gpu0
(op=33 device=5 selected=vt-native registered=1). All five numeric
guarantees are killed by the gate rather than by the compiler — rc job
a77aa104 re-expressed the three that -Werror,-Wunused-* had killed, and the
test caught all three.
So the DENSE-step refusal is gone. That is not the same as generating.
What is open
No token has been observed on any tree carrying 6b97a6800 (#2511).
The leading explanation was the allocator: #2511 narrowed managed allocation to
PageableMemoryAccess == 1, gfx1151 reports 0, and the run that last emitted
Paris, which is predates it. The prediction was that a leg under
VT_ROCM_MANAGED_ALLOC=1 — which the refusal message itself names — would
generate.
That prediction is refuted. rc job a77aa104 (strix:gpu0, tree
56fd248e9, 3 h timeout) ran exactly that leg:
LEG[managed] rc=0 wall=4118s
[ 4118s] [vt op-provider] op=4 device=5 selected=vt-native ...
[ 4118s] vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1)
[ 4118s] timeout: the monitored command dumped core
The load completed and the forward had begun; the process then died on a
core-producing signal at 4118 s, well inside its 10800 s budget. The rc=0 is
not that process's status — the runner read $? after
( … | awk … ) > log, which is awk's. The verdict is the timeout line.
So the crash is inside the forward, and the managed allocator is not the
way back to a token. That is consistent with #2511's own measurement of 17 GPU
faults in 21 managed legs against 0 in 21 without, and it is why that flag
stays a diagnostic lever and never a shipping default.
What is owed
-
The default-allocator control leg. Job
a77aa104's LEG B never ran: the
CIFS share dropped after leg A (error reading input file: Host is down—
bashreads a script lazily by byte offset and the script lived on
/workspace), and the job idled to its 8 hmax_runtimekill. There is
still no A/B, so the allocator is not isolated from the kernels. A rerun
stages its own script into/tmpfirst, as it already stages the source. -
Separate a board fault from a kernel fault. #2546 measured 12/12 GPU
resets on this board under a gate-sized native run, so "the forward crashed"
does not yet name which. A core file,dmesg, or anAMD_SERIALIZE_KERNEL=3
leg is the next traceable step. -
The ~41 min load under plain
hipMallocis still an observation, not a
measurement.a77aa104carried a per-line timestamper meant to settle it
and the timestamper did not work:vllm-cliblock-buffers behind a pipe, so
every line — the first line of the load included — reachedawkat flush and
was stamped[ 4118s]. Defeat the buffering at the source (stdbuf -oL, or
a pty) rather than stamping downstream of it. -
kDsaIndexerLogits+kDsaTopkSelecton ROCm (#2715 W2, closed with the
W1 half). Not generation-blocking on a dense step; a SPARSE step — a prompt
pastindex_topk, whererun_indexerturns true
(mla_attention.cpp:493-494) — still refuses. A benchmark workload drives
sparse steps, so this blocks any Strix speed number even once generation
returns.
Why this matters
The ROCm GLM-5.3 speed axis stays VOID while this is open, and it is the
blocker on the "at least as fast as vLLM on Strix Halo" goal for this model.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with .agents/specs/rocm-mla-attention-nonflash.md and src/vt/rocm/rocm_mla_attn.hip. Rerun the default-allocator control with the script staged in /tmp, then collect a core file, dmesg, or an AMD_SERIALIZE_KERNEL=3 trace to separate board and kernel faults. Check mla_attention.cpp:493-494 for the sparse-step path and fix the forward crash and remaining sparse-step refusal so generation and benchmarking can complete.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100