Lightning-AI / Lightning-AI/lightning-thunder
`test_networks::test_checkpointing_thunderfx` fails on (G)B200 due to grads mismatch
@kiya00 is already working on this.
Since Sep 30, 2025.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
`test_networks.py::test_checkpointing_thunderfx` fails due to grads mismatch between eager pytorch and thunderfx.
### To Reproduce
Steps to reproduce the behavior:
1. Run `test_networks.py::test_checkpointing_thunderfx`
1. See grad mismatch e.g.
```
> assert_close(grads_res, grads_ref, atol=1e-3, rtol=1e-3)
E AssertionError: Tensor-likes are not close!
E
E Mismatched elements: 62 / 20480 (0.3%)
E Greatest absolute difference: 9818.546875 at index (1, 7) (up to 0.001 allowed)
E Greatest relative difference: 22.85102081298828 at index (1, 6) (up to 0.001 allowed)
E
E The failure occurred for item [1]
```
### Expected behavior
### Environment
pjnl-20250926
### Additional context
- It seems that the test case has been failing since mid August (b: 8/13, gb: 8/21)
- The pytorch checkpointing function itself seems stable, from the file's commit history -- https://github.com/pytorch/pytorch/commits/viable/strict/torch/distributed/algorithms/_checkpoint/checkpoint_wrapper.py
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.