Lightning-AI / Lightning-AI/lightning-thunder
Support thunder for selected models and enable running the evaluation script with thunder
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
Evaluation script developed in #2592
NOTE:
* At this stage, it should be ok to support thunder with BF16.
* We don't need to support the entire model but target specific regions like MLP and MoE (as sglang already has good implementation for SDPA, verify this)
* sglang has it's own definition of the models ([llama4](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/llama4.py), [qwen3](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/qwen3_next.py), etc). We need to verify that these work with thunder and if required make fixes in the relevant repository.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the evaluation script developed in #2592, then inspect the sglang model definitions for llama4.py and qwen3_next.py. Verify which selected model regions, especially MLP and MoE, work with Thunder using BF16, and check SDPA support before making any needed fixes. Done means the evaluation script runs with Thunder for the targeted models and regions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- compilers, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100