microsoft / microsoft/onnxruntime

[ORT-nightly]Float8_e4m3 results are off-by-one

Open
#16,938 7 comments 0 reactions 1 assignee Claimed by @xadupre View on GitHub
ep:CUDA
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

Checked with(tried both gpu and cpu):
`pip install ort-nightly-gpu==1.16.0.dev20230729001 --extra-index-url=https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/ORT-Nightly/pypi/simple/`

Running a simple model that consists of the following:

input -> QuantizeLinear(scale=1, zero_point=Float8_e4m3) -> DequantizeLinear(scale=1, zero_point=Float8_e4m3) -> output

Generates what seems to be an off-by-one results(in respect of fp8 representable values).

For example:
when input==`1.8131605`, ORT produce `1.75`, while the expected result is `1.875`
when looking at the abs distance between the input and the results:
```
abs(1.875-1.8131605) = 0.06183950000000005
abs(1.8131605-1.75) = 0.06316049999999995
```

Hence we expect the the result would be 1.875, which is the correct round-to.

Adding onnx model, and numpy arrays for repro.

### To reproduce

Will add model and inputs

### Urgency

_No response_

### Platform

Linux

### OS Version

ubuntu20.04

### ONNX Runtime Installation

Other / Unknown

### ONNX Runtime Version or Commit ID

1.16.0.dev20230729001

### ONNX Runtime API

Python

### Architecture

X64

### Execution Provider

Default CPU, CUDA

### Execution Provider Library Version

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.