Fine-Grained Quantization for NPU MatMul
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 14
- Forks
- 2
- Avg merge
- 12h 42m
- Merged PRs (30d)
- 61
Description
To push inference speeds (e.g., Llama 2) to the absolute maximum and effectively saturate the NPU, we need to implement fine-grained quantization strategies at the compiler level. Currently, most operations default to unquantized f32 or bf16.
Objective:
Implement targeted quantization (int8, fp16) specifically optimized for Accelerator Topologies (like Apple ANE).
Tasks:
- Add quantization hints to the type system or as
#pragmaequivalent markers. - Implement quantization passes to dynamically convert matrices to
int8before passing them to the accelerator. - Ensure proper dequantization scaling happens transparently.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the compiler type system and MatMul lowering for accelerator topologies, then identify where quantization hints and conversion passes belong. Done means targeted int8 or fp16 quantization reaches the accelerator and dequantization scaling remains transparent, including coverage for the stated inference path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- compilers, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100