🐛 CI failure: gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs — TE 2.15.0 removed FP8GlobalStateManager.set_skip_fp8_weight_update_tensor
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Describe the bug**
CI test `gpt/gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs` failed in job [`gpt/gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs - latest`](https://github.com/NVIDIA/Megatron-LM/actions/runs/25522370262/job/75063470803?pr=4682). Tag the [@mcore-oncall](https://github.com/orgs/NVIDIA/teams/mcore-oncall) to get oncall's attention to this issue.
The failure is a hard `AttributeError` raised at the start of CUDA-graph capture, on every rank. It is triggered by the TransformerEngine 2.15.0 bump in PR #4682: TE 2.15.0 removed `FP8GlobalStateManager.set_skip_fp8_weight_update_tensor`, but `megatron/core/transformer/cuda_graphs.py` still calls it in three places (lines 611, 741, 747). The static-inference + FP8 + CUDA-graphs path goes through line 741 and crashes immediately.
**Failing run**
| Field | Value |
|-------|-------|
| PR | [#4682: Update transformer-engine dependency to version 2.15.0](https://github.com/NVIDIA/Megatron-LM/pull/4682) |
| Run | [25522370262](https://github.com/NVIDIA/Megatron-LM/actions/runs/25522370262) |
| Job | [gpt/gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/25522370262/job/75063470803?pr=4682) |
**Error**
```
[rank0]: Traceback (most recent call last):
[rank0]: File "/opt/megatron-lm/examples/inference/gpt/gpt_static_inference.py", line 249, in
[rank0]: main()
[rank0]: File "/opt/megatron-lm/examples/inference/gpt/gpt_static_inference.py", line 157, in main
[rank0]: inference_engine.generate(...)
[rank0]: File "/opt/megatron-lm/megatron/core/inference/engines/static_engine.py", line 378, in run_engine
[rank0]: self.controller.generate_all_output_tokens_static_batch(...)
[rank0]: ...
[rank0]: File "/opt/megatron-lm/megatron/core/transformer/transformer_block.py", line 822, in forward
[rank0]: hidden_states, context = layer(...)
[rank0]: File "/opt/megatron-lm/megatron/core/transformer/module.py", line 347, in __call__
[rank0]: return self.cudagraph_manager(self, args, kwargs)
[rank0]: File "/opt/megatron-lm/megatron/core/transformer/cuda_graphs.py", line 1620, in __call__
[rank0]: runner = self.get_cudagraph_runner(...)
[rank0]: File "/opt/megatron-lm/megatron/core/transformer/cuda_graphs.py", line 1549, in get_cudagraph_runner
[rank0]: runner = _CudaGraphRunner(...)
[rank0]: File "/opt/megatron-lm/megatron/core/transformer/cuda_graphs.py", line 741, in __init__
[rank0]: FP8GlobalStateManager.set_skip_fp8_weight_update_tensor(False)
[rank0]: AttributeError: type object 'FP8GlobalStateManager' has no attribute 'set_skip_fp8_weight_update_tensor'
```
All 8 ranks hit the same `AttributeError`; failure is deterministic, not flaky.
**Affected call sites in `megatron/core/transformer/cuda_graphs.py`** (any FP8 + CUDA-graph path is at risk under TE ≥ 2.15.0):
- line **611**: `FP8GlobalStateManager.set_skip_fp8_weight_update_tensor(not is_first_microbatch)`
- line **741**: `FP8GlobalStateManager.set_skip_fp8_weight_update_tensor(False)` ← triggered by this test
- line **747**: `FP8GlobalStateManager.set_skip_fp8_weight_update_tensor(False)`
**Steps/Code to reproduce bug**
Re-run the failing CI job linked above, or locally inside the dev container against TE 2.15.0:
```bash
bash tests/functional_tests/shell_test_utils/run_ci_test.sh \
--test-case gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs \
--model gpt --environment dev --platform dgx_h100
```
**Suggested fix**
Adopt the new TE 2.15.0 API at all three call sites in `megatron/core/transformer/cuda_graphs.py`. Check the TE 2.15.0 changelog / migration notes for the replacement helper (likely a renamed method or a new context-manager-style API). If the new API is conditionally available, gate on `hasattr(FP8GlobalStateManager, "set_skip_fp8_weight_update_tensor")` for backward compatibility with TE 2.14.x.
**Test is temporarily disabled**
The test scope has been suffixed with `-broken` in `tests/test_utils/recipes/h100/gpt-static-inference.yaml` to keep the TE-bump PR unblocked. Please re-enable (drop the `-broken` suffix) as part of the fix PR.
**Additional context**
Triaged automatically via `/create-issue`.
Contributor guide
Assessment
This issue has not been assessed yet.