GemmWithKReduction parallel split-K: workspace initialization is skipped, partials land in the user output buffer
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 10.5k
- Forks
- 2.1k
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 7
Description
Description
Since CUTLASS 3.2, GemmUniversalBase::initialize() only calls into Params::init_workspace when the mode is plain GEMM:
// include/cutlass/gemm/device/gemm_universal_base.h:415-420
// Assign and prepare workspace memory
if (args.mode == GemmUniversalMode::kGemm) {
return params_.init_workspace(workspace, stream);
}
return Status::kSuccess;
GemmWithKReduction overrides Params::init_workspace to do its work exactly in the other mode:
// include/cutlass/gemm/kernel/gemm_with_k_reduction.h (init_workspace)
if (this->mode == GemmUniversalMode::kGemmSplitKParallel) {
ptr_D = workspace;
ptr_gemm_k_reduction = static_cast<uint8_t *>(workspace)
+ sizeof(ElementC) * size_t(this->batch_stride_D) * size_t(this->grid_tiled_shape.k());
return Status::kSuccess;
}
The override can therefore never run: in kGemmSplitKParallel mode the base class skips the call entirely. Consequences for device::GemmWithKReduction with --parallel-split-k (example 23):
- The mainloop writes unreduced per-slice partial products through
ptr_D, which still points at the user's D buffer; with a grid whose K extent exceeds the user D allocation this writes out of bounds. - The separate reduction launch reads
ptr_gemm_k_reductionfrom a never-initialized workspace.
get_workspace_size does reserve the right size in that mode, so the buffer exists and initialize() returns success; only the redirect is skipped.
Verified statically by reading both functions; additionally exercised on hardware via a probe subclass of the protected base (example 23 flow with --parallel-split-k): after a successful initialize(), params_.ptr_D still equals the user pointer while the reported workspace size is nonzero.
Suggested fix
Widen the gate in gemm_universal_base.h so the virtual init_workspace runs for every mode (the kernels' own overrides already no-op where nothing is needed), or special-case kGemmSplitKParallel alongside kGemm.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with include/cutlass/gemm/device/gemm_universal_base.h around the workspace initialization gate, then compare it with Params::init_workspace in include/cutlass/gemm/kernel/gemm_with_k_reduction.h. Exercise the example 23 --parallel-split-k flow or the reported probe and verify that initialization redirects partial output into workspace storage and the reduction reads initialized workspace data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100