Lightning-AI / Lightning-AI/lightning-thunder

Move amax and scale update later in TEv2 grad transform

Open
#2,407 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement TransformerEngine
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature

This is an improvement idea for TEv2, if we were to join the forward call's amax and scale update with the backward, there might not be the need for the hidden checkpointing specific transform [here](https://github.com/Lightning-AI/lightning-thunder/blob/5c18b33917586e388e9526eb28cfd48cf8755915/thunder/__init__.py#L549-L552) as there would not be the need to move the checkpointed forward's amax and scale update in the backward anymore.

The new grad transform form [here](https://github.com/Lightning-AI/lightning-thunder/blob/5c18b33917586e388e9526eb28cfd48cf8755915/thunder/executors/transformer_engine_v2ex_impl.py#L202-L248) would look something like this:

```python
def _te_linear_grad_transform(a, w, bias):
[ ... ]

primal, quantized_a, quantized_w = _te_linear_fwd(a, w, bias, input_quantizer, weight_quantizer)

# No more amax scale and update here

grad_out = get_grad(primal)

backward_recipe_state = _get_te_fp8_state(recipe, "backward", 1)

(grad_output_quantizer,) = _get_te_fp8_quantizers(backward_recipe_state, 1)

grad_a, grad_w = _te_linear_bwd(
grad_out,
quantized_a,
quantized_w,
input_quantizer,
weight_quantizer,
grad_output_quantizer,
)

grad_a, grad_w = _te_fp8_amax_and_scale_update(
recipe,
# added forward state here
states=(forward_recipe_state, backward_recipe_state,),
tokens=(grad_a, grad_w),
)

[ ... ]

return primal
```

Another potential benefit would be a simpler [TransformerEngineTransformV2](https://github.com/Lightning-AI/lightning-thunder/blob/5c18b33917586e388e9526eb28cfd48cf8755915/thunder/executors/transformer_engine_v2ex_impl.py#L282C7-L282C35)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read thunder/executors/transformer_engine_v2ex_impl.py, especially _te_linear_grad_transform and TransformerEngineTransformV2, then inspect the checkpoint-specific transform in thunder/__init__.py around the linked lines. Compare where forward and backward amax/scale updates occur; done means the update moves into the gradient transform without requiring that hidden checkpointing transform and the TEv2 behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.