microsoft / microsoft/onnxruntime
[Feature Request] FP16 support for MatMul and GEMM on CPU execution provider
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
Requesting FP16 support for MatMul and GEMM on CPU execution provider.
GEMM:
Looks like GEMM use to be supported in FP16 but was disabled here: https://github.com/microsoft/onnxruntime/pull/19744
I'm not sure if I understand the motivation fully to disable. From head post, it sounds like it was driven by performance? I have an application where I want to evaluate the accuracy of using FP16. For my situation , I'm not so concerned about performance.
Matmul:
Matmul occurs often in many transformer networks. This would be a great addition to support fp16.
### Describe scenario use case
Useful to keep FP16 ops even if performance is worse if you are evaluating accuracy of fp16 vs fp32 versions of a network. It is important for me to make sure the ops are done in FP16 arithmetic and not internally casted to fp32.
Contributor guide
Research direction
Start by reviewing the CPU execution provider's MatMul and GEMM implementations, then read the linked pull request 19744 to understand why GEMM FP16 support was disabled. Done means both operators support FP16 arithmetic on the CPU execution provider without internally casting to FP32, with behavior suitable for evaluating FP16 accuracy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100