NVIDIA / NVIDIA/TransformerEngine
[MX] Rowwise `W` cached for backwards
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.5k
- Forks
- 831
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 65
Description
Describe the bug
In transformer_engine.pytorch.Linear, both rowwise and columnwise quantized W are saved for backwards.
However, only columnwise W is needed for backwards for mx mixed precision (if not caching the weight for grad accumulation or microbatching).
Notably, rowwise is discarded and only columnwise is saved in the newer transformer_engine.pytorch.ops.BasicLinear.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in transformer_engine/pytorch/module/linear.py at the Linear implementation and compare its saved backwards state with transformer_engine/pytorch/ops/basic/basic_linear.py around BasicLinear. Trace the MX mixed-precision path and identify the rowwise W state that is saved unnecessarily. Done means the legacy path matches the newer behavior without breaking weight-gradient accumulation or microbatching.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100