Benchmark Hugging Face models via Optimum integration
Open
@guangy10 is already working on this.
Since Feb 6, 2025.
module: benchmark
triaged
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
🚀 The feature, motivation and pitch
Model sourcing: huggingface/transformers
Recipes (pte generation and distribution): optimum-executorch
This work is critical as we start adding and validating recipes for non-cpu delegates.
Note this task is to track the infra work. Individual model and delegate enablement is out of scope and will be tracked separately.
Alternatives
No response
Additional context
No response
RFC (Optional)
No response
cc @huydhn @kirklandsign @shoumikhin
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.