NVIDIA / NVIDIA/TransformerEngine

Improve readability of dgrad_reduce_scatter_overlap variable name

Open
#3,164 1 comment 0 reactions 1 assignee View on GitHub

@denera is already working on this.

Since Jul 8, 2026.

Dominant language
Python
Stars
3.5k
Forks
831
Avg merge
3d 11h
Merged PRs (30d)
65

Description

The variable dgrad_reduce_scatter_overlap = ["qkv_dgrad", "fc1_dgrad"] is misleadingly named. By default both layers are placed in layers_all_gather_overlap with method="bulk". They are only moved to layers_reduce_scatter_overlap under two specific conditions:

The user explicitly provides a non-bulk method for these layers via ub_cfgs
with_cublasmp=True is set, since cuBLASMp doesn't support bulk overlap

Why the name is misleading:-
The name implies these dgrad layers always use reduce-scatter overlap, but the default is the opposite -- they default to all-gather overlap. What actually makes them special is that they are the only two GEMMs in the transformer block that support communication on both sides (all-gather on input, reduce-scatter on output), making their overlap strategy switchable -- unlike every other GEMM which has only one comm direction.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.