microsoft / microsoft/onnxruntime
[Feature Request] QMoE: support 2-bits quantized expert Weights
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
## Summary
Extend the `QMoE` (Quantized Mixture-of-Experts) operator to support 2-bit quantized expert weights, analogous to the existing 4-bit and 8-bit support.
## Requested Changes
1. **Op schema update** — add `expert_weight_bits = 2` as a valid value for the `QMoE` operator attribute.
2. **Kernel support** — implement 2-bit unpacking in the `QMoE` CUDA/CPU kernels. Expert weight tensors are packed as 4 x 2-bit values per byte (LSB-first), consistent with the `MatMulNBits` 2-bit packing convention.
### Describe scenario use case
## Motivation
Future models will use 2-bit QAD quantization (via Quark) for their dense layers. As MoE architectures become common at the frontier, 2-bit quantization of expert weights is a natural next step to enable larger MoE models to fit in memory while maintaining acceptable quality.
Currently, `QMoE` in ONNX Runtime only supports `expert_weight_bits` of 4 or 8.
Contributor guide
Research direction
Start with the QMoE operator schema and its CUDA/CPU kernels, then compare their existing 4-bit and 8-bit handling with the MatMulNBits 2-bit packing convention. The work is done when expert_weight_bits=2 is accepted and 4×2-bit, LSB-first weights are correctly unpacked in both kernel paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100