Lightning-AI / Lightning-AI/lightning-thunder

Curious about the torch.compile baseline in benchmark_litgpt

Open
#123 3 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

Was curious to know why the [`reduce-overhead` option was not used as a baseline](https://github.com/Lightning-AI/lightning-thunder/blob/3bde9e695385ba37fb365c5ed1cd1cc71995aba2/thunder/benchmarks/benchmark_litgpt.py#L261) and if there's a comparison somewhere using `reduce-overhead` ?

Thanks !

cc @apaz-cli

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at thunder/benchmarks/benchmark_litgpt.py around line 261 and inspect how the torch.compile baseline is configured. Check whether a reduce-overhead comparison already exists; done would be a documented explanation of the baseline choice and, if absent, a reproducible comparison.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.