Lightning-AI / Lightning-AI/lightning-thunder
When comparing Thunder Torch Executor to Torch Eager, the ResNet18 gradients are not close for FP32.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
*Note*: If you have a model or program that is not supported yet but should be, please use the program coverage template.
## 🐛 Bug
### To Reproduce
Steps to reproduce the behavior:
1. modify the test case
https://github.com/Lightning-AI/lightning-thunder/blob/6320b2f0cad03dff49c5141b6731587451711a4d/thunder/tests/test_inplace_functionalization.py#L184
to
```if train and executor == TorchExecutor: ```
2. Run `pytest thunder/tests/test_inplace_functionalization.py -k test_parse_resnet18_torch_cuda_float32[True]`
see error:
```
if train and executor == TorchExecutor: # and dtype == thunder.float64:
torch_grads = torch.autograd.grad(out1, ref_model.parameters(), torch.ones_like(out1))
thunder_grads = torch.autograd.grad(out2, jitted.parameters(), torch.ones_like(out2))
> torch.testing.assert_close(torch_grads, thunder_grads)
E AssertionError: Tensor-likes are not close!
E
E Mismatched elements: 9405 / 9408 (100.0%)
E Greatest absolute difference: 0.09205560386180878 at index (4, 1, 5, 0) (up to 1e-05 allowed)
E Greatest relative difference: 10.715060234069824 at index (39, 1, 3, 0) (up to 1.3e-06 allowed)
E
E The failure occurred for item [0]
thunder/tests/test_inplace_functionalization.py:187: AssertionError
=================================================== short test summary info ===================================================
FAILED thunder/tests/test_inplace_functionalization.py::test_parse_resnet18_torch_cuda_float32[True] - AssertionError: Tensor-likes are not close!
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with thunder/tests/test_inplace_functionalization.py around the ResNet18 test at line 184 and run the provided pytest command for the CUDA FP32 training case. Compare the Torch Eager and Thunder gradient results at the failing assertion; done means the test passes with gradients close enough for the existing assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100