NVIDIA / NVIDIA/cutlass

GEMM profiler verification compares only batch 0 when batch-count > 1

Open
#3,511 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

CUTLASS C++
Dominant language
C++
Stars
10.5k
Forks
2.1k
Avg merge
3d 11h
Merged PRs (30d)
7

Description

Description

The GEMM profiler's verification compares only the first batch when --batch-count=N with N > 1.

All gemm-family verify paths pass the per-batch stride as the element count:

results_.back().verification_map[library::Provider::kCUBLAS] = compare_tensors(
  options,
  *gemm_workspace_.Computed,
  *gemm_workspace_.Reference,
  gemm_workspace_.Computed->batch_stride()      // <-- elements in ONE batch
);

(tools/profiler/src/gemm_operation_profiler.cu lines ~1369-1374 for cuBLAS and ~1542-1547 for the host reference; the same pattern appears in tools/profiler/src/block_scaled_gemm_operation_profiler.cu ~1357-1370 and blockwise_gemm_operation_profiler.cu ~1195-1200.)

DeviceAllocation allocates capacity_ = batch_stride_ * batch_count_ (tools/profiler/src/device_allocation.cu ~350-356), so Computed/Reference hold all batches but compare_tensors(..., count) (tools/profiler/src/operation_profiler.cu ~630) only walks batch_stride() elements, i.e. batch 0. Batches 1..N-1 are never compared against either reference provider.

Consequence: a kernel bug that corrupts only later batches (wrong batch-stride advance in an epilogue or TMA descriptor) reports Passed under --verification=true --batch-count=N.

For contrast, the rank_k/rank_2k/symm/trmm/conv profilers compare with the full capacity, which is the conservative direction.

Suggested fix

Either loop over batches comparing each batch_stride() slice at its offset, or compare the full capacity after ensuring padding regions are written deterministically. A one-line interim hardening is to compare Computed->capacity() instead of batch_stride() when batch_count > 1.

This is a static trace through allocation, stride computation, and the comparison call; I did not construct a deliberately-wrong kernel to demonstrate a false pass.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at compare_tensors in tools/profiler/src/operation_profiler.cu and trace the verification calls around lines 1369-1374 and 1542-1547 of gemm_operation_profiler.cu. Compare the corresponding batch handling in block_scaled_gemm_operation_profiler.cu and blockwise_gemm_operation_profiler.cu, alongside DeviceAllocation capacity in device_allocation.cu. Done means verification covers every allocated batch for GEMM-family paths and can detect an error outside batch 0.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
performance, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.