microsoft / microsoft/onnxruntime

Pad behavior is inconsistent with ONNX ReferenceEvaluator and documentation

Open
#21,037 2 comments 0 reactions 1 assignee Claimed by @gramalingam View on GitHub
converter:dynamo core runtime
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

# behavior
In mode 'reflect', the result from onnxscript and onnxruntime is inconsistent with onnx.reference.ReferenceEvaluator, [onnx documentation](https://onnx.ai/onnx/operators/onnx__Pad.html), and numpy.pad.

In the eager mode of onnxscript (`res_onnxscript2`), the computation is indeed conduct by onnxruntime, right? But is the `res_onnxscript1` from onnxscript evaluator? I don't know much about onnxscript internal. But the result from onnxruntime is defenitely inconsistent, so I also created an issue there.

# code to reproduce
```python
import numpy as np
from onnxscript import script, FLOAT, opset20 as op
import onnx

data = np.array([
[1.0, 1.2],
[2.3, 3.4],
[4.5, 5.7],
], dtype = np.float32)

mode = 'reflect'

res_np = np.pad(data, ((0, 0), (2, 0)), mode = mode)
res_onnxscript1 = op.Pad(data, pads = [0, 2, 0, 0], mode = mode)

@script()
def test(data: FLOAT[None, None]) -> FLOAT[None, None]:
return op.Pad(data, pads = [0, 2, 0, 0], mode = mode)
res_onnxscript2 = test(data)

model = test.to_model_proto()
feeds = {'data': data,}

from onnx.reference import ReferenceEvaluator
sess = ReferenceEvaluator(model)
outpus_onnx = sess.run(None, feeds)

from onnxruntime import InferenceSession
sess = InferenceSession(model.SerializeToString())
outpus_onnxruntime = sess.run(None, feeds)

print(f'numpy:\n{res_np}\n')
print(f'onnxscript evaluator:\n{res_onnxscript1}\n')
print(f'onnxscript evaluator (called onnxruntime?):\n{res_onnxscript2}\n')
print(f'onnx ReferenceEvaluator:\n{outpus_onnx[0]}\n')
print(f'onnxruntime:\n{outpus_onnxruntime[0]}\n')

print(onnx.printer.to_text(model))
```
# result
```
numpy:
[[1. 1.2 1. 1.2]
[2.3 3.4 2.3 3.4]
[4.5 5.7 4.5 5.7]]

onnxscript evaluator:
Tensor(array([[0. , 1.2, 1. , 1.2],
[0. , 3.4, 2.3, 3.4],
[0. , 5.7, 4.5, 5.7]], dtype=float32))

onnxscript evaluator (called onnxruntime?):
[[0. 1.2 1. 1.2]
[0. 3.4 2.3 3.4]
[0. 5.7 4.5 5.7]]

onnx ReferenceEvaluator:
[[1. 1.2 1. 1.2]
[2.3 3.4 2.3 3.4]
[4.5 5.7 4.5 5.7]]

onnxruntime:
[[0. 1.2 1. 1.2]
[0. 3.4 2.3 3.4]
[0. 5.7 4.5 5.7]]

<
ir_version: 9,
opset_import: ["" : 20]
>
test (float[?,?] data) => (float[?,?] return_val) {
const = Constant ()
return_val = Pad (data, const)
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.