Lightning-AI / Lightning-AI/lightning-thunder
Double first iteration time for `Llama-2-7b-hf` after nvFuser direct bindings
@rdspring1 is already working on this.
Since Oct 29, 2025.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
As per title, the following command takes double the time for the first iteration at commit 8b542cf16e12cf04f6375691621bb3adb0c4acea :
```
python thunder/benchmarks/benchmark_litgpt.py --model_name Llama-2-7b-hf --compile thunder_inductor_cat --checkpoint_activations False --low_precision_mode none --micro_batch_size 1 --global_batch_size 64 --use_sdpa False --block_size 4096 --max_iters 10 --warmup_iters 5
```
compared to before #2502 :
```python
# Before
iter 0: loss 0.1650, iter time: 105627.25ms, t: 4096
# After
iter 0: loss 0.1650, iter time: 215099.65ms, t: 4096
```
Tested on the latest container on B200
cc @rdspring1 @kshitij12345 @crcrpar
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.