NVIDIA / NVIDIA/Megatron-LM

🐛 CI failure: gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs — TE 2.15.0 removed FP8GlobalStateManager.set_skip_fp8_weight_update_tensor

Open
#4,707 1 comment 0 reactions 1 assignee Claimed by @santhnm2 View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**

CI test `gpt/gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs` failed in job [`gpt/gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs - latest`](https://github.com/NVIDIA/Megatron-LM/actions/runs/25522370262/job/75063470803?pr=4682). Tag the [@mcore-oncall](https://github.com/orgs/NVIDIA/teams/mcore-oncall) to get oncall's attention to this issue.

The failure is a hard `AttributeError` raised at the start of CUDA-graph capture, on every rank. It is triggered by the TransformerEngine 2.15.0 bump in PR #4682: TE 2.15.0 removed `FP8GlobalStateManager.set_skip_fp8_weight_update_tensor`, but `megatron/core/transformer/cuda_graphs.py` still calls it in three places (lines 611, 741, 747). The static-inference + FP8 + CUDA-graphs path goes through line 741 and crashes immediately.

**Failing run**

| Field | Value |
|-------|-------|
| PR | [#4682: Update transformer-engine dependency to version 2.15.0](https://github.com/NVIDIA/Megatron-LM/pull/4682) |
| Run | [25522370262](https://github.com/NVIDIA/Megatron-LM/actions/runs/25522370262) |
| Job | [gpt/gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs - latest](https://github.com/NVIDIA/Megatron-LM/actions/runs/25522370262/job/75063470803?pr=4682) |

**Error**

```
[rank0]: Traceback (most recent call last):
[rank0]: File "/opt/megatron-lm/examples/inference/gpt/gpt_static_inference.py", line 249, in
[rank0]: main()
[rank0]: File "/opt/megatron-lm/examples/inference/gpt/gpt_static_inference.py", line 157, in main
[rank0]: inference_engine.generate(...)
[rank0]: File "/opt/megatron-lm/megatron/core/inference/engines/static_engine.py", line 378, in run_engine
[rank0]: self.controller.generate_all_output_tokens_static_batch(...)
[rank0]: ...
[rank0]: File "/opt/megatron-lm/megatron/core/transformer/transformer_block.py", line 822, in forward
[rank0]: hidden_states, context = layer(...)
[rank0]: File "/opt/megatron-lm/megatron/core/transformer/module.py", line 347, in __call__
[rank0]: return self.cudagraph_manager(self, args, kwargs)
[rank0]: File "/opt/megatron-lm/megatron/core/transformer/cuda_graphs.py", line 1620, in __call__
[rank0]: runner = self.get_cudagraph_runner(...)
[rank0]: File "/opt/megatron-lm/megatron/core/transformer/cuda_graphs.py", line 1549, in get_cudagraph_runner
[rank0]: runner = _CudaGraphRunner(...)
[rank0]: File "/opt/megatron-lm/megatron/core/transformer/cuda_graphs.py", line 741, in __init__
[rank0]: FP8GlobalStateManager.set_skip_fp8_weight_update_tensor(False)
[rank0]: AttributeError: type object 'FP8GlobalStateManager' has no attribute 'set_skip_fp8_weight_update_tensor'
```

All 8 ranks hit the same `AttributeError`; failure is deterministic, not flaky.

**Affected call sites in `megatron/core/transformer/cuda_graphs.py`** (any FP8 + CUDA-graph path is at risk under TE ≥ 2.15.0):

- line **611**: `FP8GlobalStateManager.set_skip_fp8_weight_update_tensor(not is_first_microbatch)`
- line **741**: `FP8GlobalStateManager.set_skip_fp8_weight_update_tensor(False)` ← triggered by this test
- line **747**: `FP8GlobalStateManager.set_skip_fp8_weight_update_tensor(False)`

**Steps/Code to reproduce bug**

Re-run the failing CI job linked above, or locally inside the dev container against TE 2.15.0:

```bash
bash tests/functional_tests/shell_test_utils/run_ci_test.sh \
--test-case gpt_static_inference_tp1_pp1_583m_fp8_cudagraphs \
--model gpt --environment dev --platform dgx_h100
```

**Suggested fix**

Adopt the new TE 2.15.0 API at all three call sites in `megatron/core/transformer/cuda_graphs.py`. Check the TE 2.15.0 changelog / migration notes for the replacement helper (likely a renamed method or a new context-manager-style API). If the new API is conditionally available, gate on `hasattr(FP8GlobalStateManager, "set_skip_fp8_weight_update_tensor")` for backward compatibility with TE 2.14.x.

**Test is temporarily disabled**

The test scope has been suffixed with `-broken` in `tests/test_utils/recipes/h100/gpt-static-inference.yaml` to keep the TE-bump PR unblocked. Please re-enable (drop the `-broken` suffix) as part of the fix PR.

**Additional context**

Triaged automatically via `/create-issue`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.