Lightning-AI / Lightning-AI/lightning-thunder
Automated Hugging Face transformers coverage testing
@k223kim is already working on this.
Since May 15, 2025.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
We want to systematically assess where we are w.r.t. covering HF transformers cases.
- we want automated e2e tests, maybe you can re-use some of #1984 for how to launch them automatically.
- we want to use the latest version of transformers,
- initially we only run foward, later we need to test both inference (incl static kvcache) and training (fw + bw),
- This is for testing + coverage reporting, not for fixing nor performance!
- Let's start with 10-ish models. (there is a list internally, but I don't know if it is public, yet)
Afterwards, we want to file issues for errors we hit and also look at whether the error messages are OK or need improvement.
Please reach out to Teja Pulagam (OSS contributor) to collaborate.
You could use the initial phases of the "decomposed" jit:
https://github.com/Lightning-AI/lightning-thunder/blob/3aa706a92092738e24a3f9dc08c96743646105e5/thunder/tests/test_core.py#L3216
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.