GEMM profiler verification compares only batch 0 when batch-count > 1
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 10.5k
- Forks
- 2.1k
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 7
Description
Description
The GEMM profiler's verification compares only the first batch when --batch-count=N with N > 1.
All gemm-family verify paths pass the per-batch stride as the element count:
results_.back().verification_map[library::Provider::kCUBLAS] = compare_tensors(
options,
*gemm_workspace_.Computed,
*gemm_workspace_.Reference,
gemm_workspace_.Computed->batch_stride() // <-- elements in ONE batch
);
(tools/profiler/src/gemm_operation_profiler.cu lines ~1369-1374 for cuBLAS and ~1542-1547 for the host reference; the same pattern appears in tools/profiler/src/block_scaled_gemm_operation_profiler.cu ~1357-1370 and blockwise_gemm_operation_profiler.cu ~1195-1200.)
DeviceAllocation allocates capacity_ = batch_stride_ * batch_count_ (tools/profiler/src/device_allocation.cu ~350-356), so Computed/Reference hold all batches but compare_tensors(..., count) (tools/profiler/src/operation_profiler.cu ~630) only walks batch_stride() elements, i.e. batch 0. Batches 1..N-1 are never compared against either reference provider.
Consequence: a kernel bug that corrupts only later batches (wrong batch-stride advance in an epilogue or TMA descriptor) reports Passed under --verification=true --batch-count=N.
For contrast, the rank_k/rank_2k/symm/trmm/conv profilers compare with the full capacity, which is the conservative direction.
Suggested fix
Either loop over batches comparing each batch_stride() slice at its offset, or compare the full capacity after ensuring padding regions are written deterministically. A one-line interim hardening is to compare Computed->capacity() instead of batch_stride() when batch_count > 1.
This is a static trace through allocation, stride computation, and the comparison call; I did not construct a deliberately-wrong kernel to demonstrate a false pass.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at compare_tensors in tools/profiler/src/operation_profiler.cu and trace the verification calls around lines 1369-1374 and 1542-1547 of gemm_operation_profiler.cu. Compare the corresponding batch handling in block_scaled_gemm_operation_profiler.cu and blockwise_gemm_operation_profiler.cu, alongside DeviceAllocation capacity in device_allocation.cu. Done means verification covers every allocated batch for GEMM-family paths and can detect an error outside batch 0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100