microsoft / microsoft/onnxruntime

[Feature Request] QMoE: support 2-bits quantized expert Weights

Open
#28,163 3 comments 0 reactions 0 assignees View on GitHub
feature request quantization
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

## Summary
Extend the `QMoE` (Quantized Mixture-of-Experts) operator to support 2-bit quantized expert weights, analogous to the existing 4-bit and 8-bit support.

## Requested Changes

1. **Op schema update** — add `expert_weight_bits = 2` as a valid value for the `QMoE` operator attribute.

2. **Kernel support** — implement 2-bit unpacking in the `QMoE` CUDA/CPU kernels. Expert weight tensors are packed as 4 x 2-bit values per byte (LSB-first), consistent with the `MatMulNBits` 2-bit packing convention.

### Describe scenario use case

## Motivation

Future models will use 2-bit QAD quantization (via Quark) for their dense layers. As MoE architectures become common at the frontier, 2-bit quantization of expert weights is a natural next step to enable larger MoE models to fit in memory while maintaining acceptable quality.

Currently, `QMoE` in ONNX Runtime only supports `expert_weight_bits` of 4 or 8.

Contributor guide

Open the contributing guide

Research direction

Start with the QMoE operator schema and its CUDA/CPU kernels, then compare their existing 4-bit and 8-bit handling with the MatMulNBits 2-bit packing convention. The work is done when expert_weight_bits=2 is accepted and 4×2-bit, LSB-first weights are correctly unpacked in both kernel paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.