microsoft / microsoft/onnxruntime

[Feature request] Support bfloat16/float8 inputs in `session.run()`

Open
#20,578 11 comments 0 reactions 0 assignees View on GitHub
api converter:dynamo feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

`session.run()` currently does not support bfloat16 inputs because numpy does not support bfloat16. Would it be possible to support the input via `np.uint16`? `session.run()` can accept an uint16 value which is the bit representation of the bfloat16 value. ORT should be able to interpret the value correctly because it knows the expected input type of the graph. Same can be done for the float8* types.

Contributor guide

Open the contributing guide

Research direction

Start at the session.run() input handling and trace how input types are checked against the graph's expected types. Verify the behavior for uint16 bit representations of bfloat16 and the float8 types, then add coverage showing that these inputs are interpreted correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, numpy, python
Domain
api, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.