Lightning-AI / Lightning-AI/lightning-thunder
Long first iteration time for `meta-llama/Llama-4-Maverick-17B-128E`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
When running `benchmark_inference.py` the following command leads to ~33 minutes for the first iteration:
```
python thunder/benchmarks/benchmark_inference.py --input-length 2048 --output-length 512 --mode thunder --num-iterations 10 --num-layers 2
```
### To Reproduce
1. Get the latest container up an running;
2. remove 13f7171784d6a953fd02879f325a7facfe124d0d commit by checking out earlier or committing inverse (to avoid #2680)
3. Run inference script with the above command
### Note
Unfortunately, the patch listed in #2677 does not help reducing compile time
[UPDATE] - It seems to be related to many recompilations
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running thunder/benchmarks/benchmark_inference.py with the command in the report and measure the first iteration and recompilations. Investigate the reported repeated recompilation behavior, using the earlier commit context and issue #2677 as references. Done means the benchmark no longer spends about 33 minutes on its first iteration, with the improvement verified by the same command.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- compilers, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100