microsoft / microsoft/onnxruntime
[MLAS Feature Request] MatMulNBits faster implementation for fp16 input dtype
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
Please add support for HQNBIT_CompInt8 computation path for MatMulNBits fp16 input datatype. Currently the performance of MatMulNBits is much slower for fp16 vs fp32 (~6x) on CPU.
### Describe scenario use case
For FP16 input dtype, MatMulNBits computation is always falling back to HQNBIT_CompFp16 as there is no implementation present for [HQNBIT_CompInt8](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/mlas/lib/qnbitgemm.cpp#L86) compute type.
Contributor guide
Research direction
Start in onnxruntime/core/mlas/lib/qnbitgemm.cpp at the HQNBIT_CompInt8 compute type, then trace how MatMulNBits dispatches fp16 input. Implement support for the fp16 input path without falling back to HQNBIT_CompFp16, and verify that the MatMulNBits CPU performance improves toward the fp32 path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100