gfx1151: establish valid rocprofv3 timestamps before kernel attribution
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-GATE-ROCM-LLAMACPP
The #3015 same-tool attribution is blocked by invalid rocprofv3 timestamps on Strix Halo. A kernel-only trace completed the requested 64-token Qwen3.8-27B Q4_K_M generation, but emitted 62 hsa_amd_profiling_get_dispatch_time warnings saying start time is after end time and swapping the values. Positive exported durations therefore do not prove valid timestamps.
Evidence: /mnt/nas_share/rc/strix-3015-20260907.Gq6MW5/diag-kernel-control-asaj36ta/combined.log and trace/e5367aefafa8/1_kernel_trace.csv. Lease 25065496-5849-4a03-820b-df55045aa9eb, 2026-09-07, rocprofiler-sdk 1.1.0 (97f5574fe2fdc7bef44fb01545347912ee9f1779), ROCm 7.2.4, kernel 6.14.0-36-generic, gfx1151. Source f98b638673b4d2edc0250eec56d229357ea38ab1; model SHA256 7e78da5d7e3ae28d178121f58646953305f3e5bd3cb46f4a75584e8b6c6fe169; image d554fe849c5e3c4d6b81227e0b87f165fa3b7b015899872bd93bc732c12ffeb6.
An earlier combined kernel/HIP/memory-copy/scratch trace in lease 43c3e415-e8e5-4918-ab42-b4d9c3dcab29 emitted 16 such warnings and then faulted after ten sampling steps. The driver automatically reset the GPU successfully. The profiler stalled in its signal handler until the runner timeout. #3039 separately repairs that delayed fault detection.
Controls after automatic recovery: HIP initialization succeeds with one device; unprofiled CLI help succeeds; an unprofiled same-binary 64-token completion succeeds; profiler-wrapped help succeeds; kernel-only 64-token completion succeeds with the 62 timestamp warnings. These controls narrow conditions but do not prove the cause of the combined-trace hang.
Required: establish a trustworthy same-tool trace on both engines before any per-kernel timing share or host-idle bound. Preserve raw warnings and invalid traces. Do not treat swapped timestamps as a repaired measurement or attribute the hang to the last recorded kernel. A tool/runtime/kernel change needs explicit provenance and a fresh run; do not automatically reset or clear the device.
Owner: BACKEND-GATE-ROCM-LLAMACPP; spec .agents/specs/rocm-strix-kernel-trace.md, under Owed. The broader correctness gate remains TOKEN_GATE=FAIL.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with .agents/specs/rocm-strix-kernel-trace.md and review the preserved combined.log and trace/e5367aefafa8/1_kernel_trace.csv artifacts. Reproduce same-tool traces on both engines while preserving raw warnings and invalid traces, with explicit tool, runtime, kernel, source, and model provenance; done requires trustworthy timestamps before timing attribution, without resetting or clearing the device.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, linux
- Domain
- observability, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100