microsoft / microsoft/onnxruntime
[Feature Request] Add int4/uint4 support for IBM Power Systems (PowerPC; VSX/MMA)
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
**Description**:
ONNX Runtime supports int4/uint4 mainly on x86 (AVX2/AVX-512), but lacks optimized support for IBM Power Systems (PowerPC). This limits efficient inference of 4-bit quantized models on this architecture.
**Proposal:**
- Add VSX + MMA optimized kernels (e.g., GEMM/MatMul, dot products)
- Extend MLAS (or equivalent) with PowerPC paths
- Support existing packed int4 formats
**Notes:**
- VSX can handle unpacking; MMA can accelerate matrix multiply/accumulation
- Approach can follow existing x86 implementations (e.g., ggml)
**Questions:**
- Any plans for non-x86 int4 support?
- Preferred integration point (MLAS vs EP)?
Thanks!
### Describe scenario use case
Running LLM inference on IBM Power Systems using 4-bit quantized models (e.g., weight-only int4). Without native int4/uint4 support in ONNX Runtime, deployments must fall back to int8 or higher precision, leading to increased memory bandwidth usage and reduced throughput.
Enabling int4 with VSX/MMA would allow efficient execution of quantized GEMM/dot-product workloads, improving performance and reducing memory footprint for large models on PowerPC-based systems.
Contributor guide
Research direction
Review the existing x86 int4 implementations and the MLAS or execution-provider integration points mentioned in the proposal. Compare their packed int4 formats and kernel responsibilities with the available VSX and MMA capabilities. Done means supported int4/uint4 inference on IBM Power Systems with optimized GEMM or dot-product paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100