Lightning-AI / Lightning-AI/lightning-thunder
[Regressions] ThunderFX is slower than 2 weeks ago for 3 models
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
Recently found regressions:

falcon-40b -> mentioned in https://github.com/Lightning-AI/lightning-thunder/issues/1366
falcon-180B -> mentioned in https://github.com/Lightning-AI/lightning-thunder/issues/1484
falcon-7b -> New, regression probably due to MBS difference w.r.t torch.compile
pythia-12b -> New, fp8 regression, bf16 regression probably due to MBS difference w.r.t torch.compile
Platypus-30B -> mentioned in https://github.com/Lightning-AI/lightning-thunder/issues/1366
Gemma-7b -> mentioned in https://github.com/Lightning-AI/lightning-thunder/issues/1484
vicuna-33b-v1.3 -> mentioned in https://github.com/Lightning-AI/lightning-thunder/issues/1366
pythia-14m -> New, regression probably due to MBS difference w.r.t torch.compile
### To Reproduce
Tested on `pjnl-20241205` (as in the Latest image date in the screenshot).
### Environment
system.device_product_name DGXH100
system.gpu_driver_version 535.129.03
libraries.cuda 12.8.33
libraries.pip.lightning 2.4.0.dev20240728
libraries.pip.lightning-thunder 0.2.0.dev0
libraries.pip.lightning-utilities 0.11.9
libraries.pip.litgpt 0.4.11
libraries.pip.nvfuser 0.2.23+git1dda106
libraries.pip.pytorch-lightning 2.4.0
libraries.pip.torch 2.6.0a0+git45ed7c1
libraries.pip.torchao NaN
libraries.pip.torchmetrics 1.6.0
libraries.pip.torchvision 0.19.0a0+d23a6e1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the listed model benchmarks on pjnl-20241205 with the reported DGXH100, CUDA, PyTorch, and Lightning Thunder versions, then compare results with the performance from two weeks earlier. Review the related issues 1366 and 1484 for existing investigations; done means identifying and resolving the regressions and confirming improved results for the affected models.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- compilers, machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100