microsoft / microsoft/onnxruntime

[MLAS Feature Request] MatMulNBits faster implementation for fp16 input dtype

Open
#27,251 2 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

Please add support for HQNBIT_CompInt8 computation path for MatMulNBits fp16 input datatype. Currently the performance of MatMulNBits is much slower for fp16 vs fp32 (~6x) on CPU.

### Describe scenario use case

For FP16 input dtype, MatMulNBits computation is always falling back to HQNBIT_CompFp16 as there is no implementation present for [HQNBIT_CompInt8](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/mlas/lib/qnbitgemm.cpp#L86) compute type.

Contributor guide

Open the contributing guide

Research direction

Start in onnxruntime/core/mlas/lib/qnbitgemm.cpp at the HQNBIT_CompInt8 compute type, then trace how MatMulNBits dispatches fp16 input. Implement support for the fp16 input path without falling back to HQNBIT_CompFp16, and verify that the MatMulNBits CPU performance improves toward the fp32 path.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.