test_backend_cross_device: causal conv1d fwd/update flakes intermittently on gfx1200
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BACKEND-ROCM
test_backend_cross_device.cpp:1459, TEST_CASE("causal conv1d fwd/update match the CPU oracle"), fails intermittently on isravale (RX 9060 XT, gfx1200, ROCm 7.2.3) with a real numerical mismatch, not a crash:
tests/vt/test_backend_cross_device.cpp:1534: ERROR: CHECK( Nmse(ref_out, dout.Download()) <= kNmseTol ) is NOT correct!
values: CHECK( 0.0858971 <= 0.0005 )
logged: DeviceName(dt) := 1
Reproduction: run the isolated case (--test-case="causal conv1d fwd/update match the CPU oracle") repeatedly under the GPU file mutex. Observed 3 failures in 8 consecutive runs (~37.5%), same NMSE value (0.0858971) both times it was captured. Not a timeout, not a launch failure — the kernel runs and returns a wrong number some fraction of the time.
Confirmed unrelated to any in-flight PR: the test case and the GDN conv1d kernel source are both untouched by row/KERNEL-QUANT-CIQ-GEMM-ROCM-IQUANT-w1's reconciliation work (verified present, unmodified, at that branch's pre-reconciliation head 22800690b). Not previously tracked under any issue search for conv1d/ROCm/gfx1200.
Given this project's stated position that no device-side ASan exists for RDNA/gfx12 and that green tests have twice this session alone hidden a real defect in adjacent ROCm kernel work (races, OOB reads), this deserves the same address-arithmetic-level scrutiny rather than being waved off as environment noise — but it is out of scope for the PR that found it, which does not touch this kernel.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/vt/test_backend_cross_device.cpp:1459 and run the isolated causal conv1d case repeatedly under the GPU file mutex. Then inspect the GDN conv1d kernel source and its address arithmetic, comparing results with the CPU oracle. Done means repeated runs stay within the Nmse tolerance on the affected gfx1200 device.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- computer-graphics, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100