Lightning-AI / Lightning-AI/lightning-thunder

Long first iteration time for `meta-llama/Llama-4-Maverick-17B-128E`

Open
#2,687 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

symbolic values thunderfx
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug

When running `benchmark_inference.py` the following command leads to ~33 minutes for the first iteration:

```
python thunder/benchmarks/benchmark_inference.py --input-length 2048 --output-length 512 --mode thunder --num-iterations 10 --num-layers 2
```

### To Reproduce

1. Get the latest container up an running;
2. remove 13f7171784d6a953fd02879f325a7facfe124d0d commit by checking out earlier or committing inverse (to avoid #2680)
3. Run inference script with the above command

### Note

Unfortunately, the patch listed in #2677 does not help reducing compile time

[UPDATE] - It seems to be related to many recompilations

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running thunder/benchmarks/benchmark_inference.py with the command in the report and measure the first iteration and recompilations. Investigate the reported repeated recompilation behavior, using the earlier commit context and issue #2677 as references. Done means the benchmark no longer spends about 33 minutes on its first iteration, with the improvement verified by the same command.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
compilers, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.