mudler / mudler/vllm.cpp

gfx1151: establish valid rocprofv3 timestamps before kernel attribution

Open
#3,040 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-GATE-ROCM-LLAMACPP

The #3015 same-tool attribution is blocked by invalid rocprofv3 timestamps on Strix Halo. A kernel-only trace completed the requested 64-token Qwen3.8-27B Q4_K_M generation, but emitted 62 hsa_amd_profiling_get_dispatch_time warnings saying start time is after end time and swapping the values. Positive exported durations therefore do not prove valid timestamps.

Evidence: /mnt/nas_share/rc/strix-3015-20260907.Gq6MW5/diag-kernel-control-asaj36ta/combined.log and trace/e5367aefafa8/1_kernel_trace.csv. Lease 25065496-5849-4a03-820b-df55045aa9eb, 2026-09-07, rocprofiler-sdk 1.1.0 (97f5574fe2fdc7bef44fb01545347912ee9f1779), ROCm 7.2.4, kernel 6.14.0-36-generic, gfx1151. Source f98b638673b4d2edc0250eec56d229357ea38ab1; model SHA256 7e78da5d7e3ae28d178121f58646953305f3e5bd3cb46f4a75584e8b6c6fe169; image d554fe849c5e3c4d6b81227e0b87f165fa3b7b015899872bd93bc732c12ffeb6.

An earlier combined kernel/HIP/memory-copy/scratch trace in lease 43c3e415-e8e5-4918-ab42-b4d9c3dcab29 emitted 16 such warnings and then faulted after ten sampling steps. The driver automatically reset the GPU successfully. The profiler stalled in its signal handler until the runner timeout. #3039 separately repairs that delayed fault detection.

Controls after automatic recovery: HIP initialization succeeds with one device; unprofiled CLI help succeeds; an unprofiled same-binary 64-token completion succeeds; profiler-wrapped help succeeds; kernel-only 64-token completion succeeds with the 62 timestamp warnings. These controls narrow conditions but do not prove the cause of the combined-trace hang.

Required: establish a trustworthy same-tool trace on both engines before any per-kernel timing share or host-idle bound. Preserve raw warnings and invalid traces. Do not treat swapped timestamps as a repaired measurement or attribute the hang to the last recorded kernel. A tool/runtime/kernel change needs explicit provenance and a fresh run; do not automatically reset or clear the device.

Owner: BACKEND-GATE-ROCM-LLAMACPP; spec .agents/specs/rocm-strix-kernel-trace.md, under Owed. The broader correctness gate remains TOKEN_GATE=FAIL.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .agents/specs/rocm-strix-kernel-trace.md and review the preserved combined.log and trace/e5367aefafa8/1_kernel_trace.csv artifacts. Reproduce same-tool traces on both engines while preserving raw warnings and invalid traces, with explicit tool, runtime, kernel, source, and model provenance; done requires trustworthy timestamps before timing attribution, without resetting or clearing the device.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
observability, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.