Lightning-AI / Lightning-AI/lightning-thunder

`vit_hf.py` fails only when NVFuser is enabled

Open
#2,103 0 comments 0 reactions 0 assignees View on GitHub
huggingface nvfuser
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

*Note*: If you have a model or program that is not supported yet but should be, please use the program coverage template.

## 🐛 Bug
`torch.testing.assert_close(out, thunder_out)` passes successfully without NVFuser. However, with nvfuser, it fails with the following error:
```
Traceback (most recent call last):
File "/teamspace/studios/this_studio/lightning-thunder/examples/quickstart/vit_hf.py", line 38, in
main()
File "/teamspace/studios/this_studio/lightning-thunder/examples/quickstart/vit_hf.py", line 29, in main
torch.testing.assert_close(out, thunder_out)
File "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/torch/testing/_comparison.py", line 1519, in assert_close
raise error_metas[0].to_error(msg)
AssertionError: Tensor-likes are not close!

Mismatched elements: 124694 / 128000 (97.4%)
Greatest absolute difference: 0.003966569900512695 at index (93, 851) (up to 1e-05 allowed)
Greatest relative difference: 9.088653564453125 at index (76, 519) (up to 1.3e-06 allowed)
```

### To Reproduce

- To install nvfuser, do
```
pip install nvfuser-cu128-torch27
```

#### Code sample

### Expected behavior

### Environment

- PyTorch Version (e.g., 1.0):
- OS (e.g., Linux):
- How you installed PyTorch (`conda`, `pip`, source):
- Build command you used (if compiling from source):
- Python version:
- CUDA/cuDNN version:
- GPU models and configuration:
- Any other relevant information:

### Additional context

cc. @IvanYashchuk @kevinstephano @csarofeen @jjsjann123 @t-vi

cc @tfogal

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with examples/quickstart/vit_hf.py, especially the torch.testing.assert_close call at line 29, and reproduce it with and without NVFuser after installing nvfuser-cu128-torch27. Compare the two outputs and compiler paths; done means the example's assertion passes when NVFuser is enabled without changing the expected model behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
compilers, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.