Lightning-AI / Lightning-AI/lightning-thunder

When comparing Thunder Torch Executor to Torch Eager, the ResNet18 gradients are not close for FP32.

Open
#655 4 comments 0 reactions 0 assignees View on GitHub
network support numerical accuracy
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

*Note*: If you have a model or program that is not supported yet but should be, please use the program coverage template.

## 🐛 Bug

### To Reproduce

Steps to reproduce the behavior:

1. modify the test case
https://github.com/Lightning-AI/lightning-thunder/blob/6320b2f0cad03dff49c5141b6731587451711a4d/thunder/tests/test_inplace_functionalization.py#L184
to
```if train and executor == TorchExecutor: ```
2. Run `pytest thunder/tests/test_inplace_functionalization.py -k test_parse_resnet18_torch_cuda_float32[True]`
see error:
```
if train and executor == TorchExecutor: # and dtype == thunder.float64:
torch_grads = torch.autograd.grad(out1, ref_model.parameters(), torch.ones_like(out1))
thunder_grads = torch.autograd.grad(out2, jitted.parameters(), torch.ones_like(out2))
> torch.testing.assert_close(torch_grads, thunder_grads)
E AssertionError: Tensor-likes are not close!
E
E Mismatched elements: 9405 / 9408 (100.0%)
E Greatest absolute difference: 0.09205560386180878 at index (4, 1, 5, 0) (up to 1e-05 allowed)
E Greatest relative difference: 10.715060234069824 at index (39, 1, 3, 0) (up to 1.3e-06 allowed)
E
E The failure occurred for item [0]

thunder/tests/test_inplace_functionalization.py:187: AssertionError
=================================================== short test summary info ===================================================
FAILED thunder/tests/test_inplace_functionalization.py::test_parse_resnet18_torch_cuda_float32[True] - AssertionError: Tensor-likes are not close!

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with thunder/tests/test_inplace_functionalization.py around the ResNet18 test at line 184 and run the provided pytest command for the CUDA FP32 training case. Compare the Torch Eager and Thunder gradient results at the failing assertion; done means the test passes with gradients close enough for the existing assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.