Sparse split-K kernels offset the E operand without the kElementsPerElementE division
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 10.5k
- Forks
- 2.1k
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 7
Description
Description
SparseGemmSplitKParallel-style serial split-K in include/cutlass/gemm/kernel/sparse_gemm.h offsets the E (metadata) operand by the wrong amount for slices beyond the first:
// sparse_gemm.h:237
cutlass::MatrixCoord tb_offset_E{
threadblock_tile_offset.m() * Mma::Shape::kM,
threadblock_tile_offset.k() * params.gemm_k_size / kSparse, // missing / kElementsPerElementE
};
The E iterator's extent is expressed in packed element-E columns and divides twice (sparse_gemm.h:269, problem_size_k / kSparse / kElementsPerElementE), and the sibling kernel gemm_sparse_universal.h:622 divides twice as well when offsetting:
cutlass::MatrixCoord tb_offset_E{
threadblock_tile_offset.m() * Mma::Shape::kM,
offset_k / kSparse / kElementsPerElementE,
};
With the missing division, every K-slice >= 1 of a split-K sparse GEMM starts its metadata reads far past the end of the packed E tensor; all accesses are predicated off and the slice runs on zero-filled (invalid) metadata.
Concrete numbers: fp16 (kElementsPerElementE == 2), K=128, split_k_slices = 2: slice 1 has an E extent of 4 columns while its offset column is 32. Zero valid accesses.
The same formula is duplicated in two more copies of this kernel body:
sparse_gemm_with_absmax.h:322sparse_gemm_with_visitor.h:153
Reachability note: nothing in-tree exercises it (device::SparseGemm with split_k_slices > 1; example 15 uses one slice), so this is latent, but the wrapper accepts split-K (device/sparse_gemm.h passes args.batch_count through).
Suggested fix
Divide by kElementsPerElementE in the offset at all three sites, matching gemm_sparse_universal.h:
threadblock_tile_offset.k() * params.gemm_k_size / kSparse / kElementsPerElementE,
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the three duplicated offset sites in include/cutlass/gemm/kernel/sparse_gemm.h, sparse_gemm_with_absmax.h, and sparse_gemm_with_visitor.h, then compare them with gemm_sparse_universal.h:622 and the iterator extent around sparse_gemm.h:269. Check how device/sparse_gemm.h passes split-K arguments and inspect example 15 for existing coverage. Done means split-K metadata offsets use the packed E-column extent consistently at all three sites.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100