microsoft / microsoft/onnxruntime

[Feature Request] FP16 support for MatMul and GEMM on CPU execution provider

Open
#22,467 0 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

Requesting FP16 support for MatMul and GEMM on CPU execution provider.

GEMM:
Looks like GEMM use to be supported in FP16 but was disabled here: https://github.com/microsoft/onnxruntime/pull/19744
I'm not sure if I understand the motivation fully to disable. From head post, it sounds like it was driven by performance? I have an application where I want to evaluate the accuracy of using FP16. For my situation , I'm not so concerned about performance.

Matmul:
Matmul occurs often in many transformer networks. This would be a great addition to support fp16.

### Describe scenario use case

Useful to keep FP16 ops even if performance is worse if you are evaluating accuracy of fp16 vs fp32 versions of a network. It is important for me to make sure the ops are done in FP16 arithmetic and not internally casted to fp32.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the CPU execution provider's MatMul and GEMM implementations, then read the linked pull request 19744 to understand why GEMM FP16 support was disabled. Done means both operators support FP16 arithmetic on the CPU execution provider without internally casting to FP32, with behavior suitable for evaluating FP16 accuracy.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.