mudler / mudler/vllm.cpp

BACKEND-ROCM: ROCGdb CFA failure blocks causal-convolution instruction capture

Open
#3,054 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ROCM

The #2773 numerical-characterization acquisition cannot complete its instruction-step gate with the approved ROCm image. ROCGdb terminates with gdb/dwarf2/frame.c:1052: internal-error: Unknown CFA rule while preparing stepi at the first selected causal-convolution BF16 DOT2. This tracks the diagnostic capability gap under #2773; it does not change the underlying failing numerical test.

The operator reproduced this on 8 September 2026 with ROCGdb 16.3, rocm-rel-7.2-90, executable SHA256 8c239fef5ae0af68f478eee1f977f242fe38d8c0ffc2ec3c3fae9d1f4408e758, in image sha256:80aab4c182a1f3eeebe286173977e57fcaf10a049b41f475655b35d285de31dc. The unchanged oracle is vLLM e126687a9a828d513c01a07cd69f025f27d63280, Triton 3.7.1+f0b55c07, HIP 7.2.53211, on gfx1100 PCI 0000:03:00.0. Existing approved device/ptrace permissions were sufficient to stop and read the kernel. No runtime, compiler flags, registers, MODE, or PC were changed.

The unchanged collector selected test_causal_conv1d_varlen[4-False-4096-8-4-True-True-itype0]. Before the crash, the actual dispatch descriptor, live memory-URI source extent, and loaded instruction bytes were joined. The copied 9,816-byte ELF exactly matched SHA256 d2f70112304de9a32fbbddc0c530ad43ec4d0fbffda0290b64195d936e016518. The target was group (0,1,0), work-item (5,0,0), with original ABI registers and live sequence/chunk/cache-row/input/weight/bias words recorded.

At entry-relative PC +0x8ec (candidate ELF VA 0x21ec), v_dot2_bf16_bf16 v18.l, v18, v10, 0 had selected-lane sources 0x00003fbc and 0x00004062, MODE 0x3f0, and EXEC 0xffffffff. stepi then failed through dwarf2_frame_cache, compute_frame_id, set_step_info, prepare_one_step, and step_1. ROCGdb/container exit was 127. There is no after-instruction value, completed collector artifact, verified breakpoint restoration, or successful debugger preservation control. The partial acquisition is explicitly refused; the original numerical result remains unresolved by it.

A fresh plain control after the crash completed normally: the expected upstream AssertionError/pytest exit1 remained, and all 67 raw/f32 tensor files matched the accepted earlier capture. The original caches and failed debugger output are preserved separately.

Upstream has a related ROCGdb report #47 and a reported GDB fix a910478f65a596817fde3ad4ee727e41ec8e0513. That report is a source lead, not proof that its exact lower-level cause or fix applies to this binary.

Completion requires a source-bound, independently reviewed acquisition method that preserves this exact runtime/kernel and records actual before/after values, live instruction/coordinate linkage, normal breakpoint restoration, and complete repeated debugger/plain controls. Preserve this failed attempt. Do not silently replace the approved debugger/runtime or accept arithmetic predictions as measurements.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the test_causal_conv1d_varlen[4-False-4096-8-4-True-True-itype0] collector described here and compare the ROCGdb failure with upstream ROCGdb report #47 and fix a910478f65a596817fde3ad4ee727e41ec8e0513. Done requires an independently reviewed acquisition on the exact runtime and kernel with before/after values, instruction and coordinate linkage, breakpoint restoration, and repeated debugger/plain controls.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
devtools
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.