NVIDIA / NVIDIA/TransformerEngine
Improve readability of dgrad_reduce_scatter_overlap variable name
@denera is already working on this.
Since Jul 8, 2026.
- Dominant language
- Python
- Stars
- 3.5k
- Forks
- 831
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 65
Description
The variable dgrad_reduce_scatter_overlap = ["qkv_dgrad", "fc1_dgrad"] is misleadingly named. By default both layers are placed in layers_all_gather_overlap with method="bulk". They are only moved to layers_reduce_scatter_overlap under two specific conditions:
The user explicitly provides a non-bulk method for these layers via ub_cfgs
with_cublasmp=True is set, since cuBLASMp doesn't support bulk overlap
Why the name is misleading:-
The name implies these dgrad layers always use reduce-scatter overlap, but the default is the opposite -- they default to all-gather overlap. What actually makes them special is that they are the only two GEMMs in the transformer block that support communication on both sides (all-gather on input, reduce-scatter on output), making their overlap strategy switchable -- unlike every other GEMM which has only one comm direction.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.