microsoft / microsoft/onnxruntime

[Feature Request] Add int4/uint4 support for IBM Power Systems (PowerPC; VSX/MMA)

Open
#28,199 0 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

**Description**:

ONNX Runtime supports int4/uint4 mainly on x86 (AVX2/AVX-512), but lacks optimized support for IBM Power Systems (PowerPC). This limits efficient inference of 4-bit quantized models on this architecture.

**Proposal:**

- Add VSX + MMA optimized kernels (e.g., GEMM/MatMul, dot products)
- Extend MLAS (or equivalent) with PowerPC paths
- Support existing packed int4 formats

**Notes:**

- VSX can handle unpacking; MMA can accelerate matrix multiply/accumulation
- Approach can follow existing x86 implementations (e.g., ggml)

**Questions:**

- Any plans for non-x86 int4 support?
- Preferred integration point (MLAS vs EP)?

Thanks!

### Describe scenario use case

Running LLM inference on IBM Power Systems using 4-bit quantized models (e.g., weight-only int4). Without native int4/uint4 support in ONNX Runtime, deployments must fall back to int8 or higher precision, leading to increased memory bandwidth usage and reduced throughput.

Enabling int4 with VSX/MMA would allow efficient execution of quantized GEMM/dot-product workloads, improving performance and reducing memory footprint for large models on PowerPC-based systems.

Contributor guide

Open the contributing guide

Research direction

Review the existing x86 int4 implementations and the MLAS or execution-provider integration points mentioned in the proposal. Compare their packed int4 formats and kernel responsibilities with the available VSX and MMA capabilities. Done means supported int4/uint4 inference on IBM Power Systems with optimized GEMM or dot-product paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.