Lightning-AI / Lightning-AI/lightning-thunder
[Regressions] ThunderFX is slower than 2 weeks ago for 2 models
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
Recently found regressions:
### To Reproduce
All parameters to `benchmark_litgpt.py` are visible in the attached image.
### Environment
Tested on `pjnl-20241122` (as in the `Latest image date` in the screenshot).
system.device_product_name DGXH100
system.gpu_driver_version 535.129.03
libraries.cuda 12.6.3.001
libraries.pip.lightning 2.4.0.dev20240728
libraries.pip.lightning-thunder 0.2.0.dev0
libraries.pip.lightning-utilities 0.11.9
libraries.pip.litgpt 0.4.11
libraries.pip.nvfuser 0.2.23+gitb5e5182
libraries.pip.pytorch-lightning 2.4.0
libraries.pip.torch 2.6.0a0+gitecf3bae
libraries.pip.torchao 0.6.1
libraries.pip.torchmetrics 1.6.0
libraries.pip.torchvision 0.19.0a0+d23a6e1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with benchmark_litgpt.py and the parameters visible in the attached reproduction screenshot, using the pjnl-20241122 environment details as the baseline. Compare the two affected model benchmarks with the earlier performance and identify what changed; done means the regression is explained and performance is restored or the remaining cause is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- compilers, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100