Benchmark quantization
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 346
- PR merge metrics
- No merged PRs in 30d
Description
Add benchmarks for quantized models.
This might be implemented as a new 'flavor' of test_eval, where most models raise NotImplemented and it is strictly opt-in to add quantization for particular models.
@jamesr66a can you add any specifics around which models you'd like to quantize and what the minimal number of quantized models is that would be useful to enable?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing test_eval benchmark flow and how benchmark flavors are defined. Clarify which quantized models and the minimum useful model count with the issue participants before implementation. Done means quantized-model benchmarks are strictly opt-in and run successfully for the agreed models.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100