Lightning-AI / Lightning-AI/lightning-thunder

Support thunder for selected models and enable running the evaluation script with thunder

Open
#2,593 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

Evaluation script developed in #2592

NOTE:
* At this stage, it should be ok to support thunder with BF16.
* We don't need to support the entire model but target specific regions like MLP and MoE (as sglang already has good implementation for SDPA, verify this)
* sglang has it's own definition of the models ([llama4](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/llama4.py), [qwen3](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/models/qwen3_next.py), etc). We need to verify that these work with thunder and if required make fixes in the relevant repository.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the evaluation script developed in #2592, then inspect the sglang model definitions for llama4.py and qwen3_next.py. Verify which selected model regions, especially MLP and MoE, work with Thunder using BF16, and check SDPA support before making any needed fixes. Done means the evaluation script runs with Thunder for the targeted models and regions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
compilers, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.