[Bug] InstanceNorm2d -> Reciprocal -> Conv2d causing mismatch between tvm and ONNXRuntime
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 4k
- Avg merge
- 2d 1h
- Merged PRs (30d)
- 112
Description
### Expected behavior
TVM and ONNXRuntime should give the same output
### Actual behavior
`––––– MISMATCH DETECTED –––––
Not equal to tolerance rtol=0.01, atol=0.001
Mismatched elements: 40 / 16384 (0.244%)
Max absolute difference: 0.12496948
Max relative difference: 1.3682545
x: array([[[[-2.420260e+00, -6.435647e+00, -7.613084e-01, ...,
-1.298009e+01, -1.387613e+01, 1.234641e+01],
[ 1.387765e+00, 5.537911e+00, 2.182506e+01, ...,...
y: array([[[[-2.420275e+00, -6.435628e+00, -7.613027e-01, ...,
-1.298006e+01, -1.387616e+01, 1.234646e+01],
[ 1.387758e+00, 5.537896e+00, 2.183493e+01, ...,...`
### Environment
TVM:0.17.0
ONNXRuntime:1.16.3
### Steps to reproduce
```python
import random
import sys, os
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
import torch
import torch.nn as nn
import torch.nn.functional as F
import tempfile
import onnx
import onnxruntime as ort
from numpy.testing import assert_allclose
import tvm
from tvm import relay
from tvm.contrib import graph_executor
import numpy as np
import nas_model_2
class SimpleBugModel(nn.Module):
def __init__(self):
super().__init__()
self.input_conv = torch.nn.modules.conv.Conv2d(in_channels=3, out_channels=16, kernel_size=1)
self.block1 = torch.nn.modules.instancenorm.InstanceNorm2d(num_features=16)
self.block2 = nas_model_2.ReciprocalWrapper()
self.block3 = torch.nn.modules.conv.Conv2d(kernel_size=5, padding=2, in_channels=16,
out_channels=16)
def forward(self, x):
__input_conv = self.input_conv(x)
__blocks__1 = self.block1(__input_conv)
__blocks__2 = self.block2(__blocks__1)
__blocks__3 = self.block3(__blocks__2)
return __blocks__3
def main():
seed=295007241
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
model = SimpleBugModel()
model.eval()
dummy = torch.randn(1, 3, 32, 32, dtype=torch.float32)
with tempfile.NamedTemporaryFile(suffix='.onnx', delete=False) as tmp:
onnx_path = tmp.name
torch.onnx.export(model, dummy, onnx_path, opset_version=19, input_names=['input'], output_names=['output'])
ort_sess = ort.InferenceSession(onnx_path, providers=['CPUExecutionProvider'])
ort_out = ort_sess.run(None, {'input': dummy.numpy()})[0]
onnx_model = onnx.load(onnx_path)
shape_dict = {'input': dummy.numpy().shape}
mod, params = relay.frontend.from_onnx(onnx_model, shape_dict, freeze_params=True)
with tvm.transform.PassContext(opt_level=4):
lib = relay.build(mod, target='llvm', params=params)
m = graph_executor.GraphModule(lib['default'](tvm.cpu()))
m.set_input('input', tvm.nd.array(dummy.numpy()))
m.run()
tvm_out = m.get_output(0)
tvm_out = tvm_out.numpy()
try:
assert_allclose(ort_out, tvm_out, rtol=1e-2, atol=1e-3, equal_nan=True)
except AssertionError as e:
print('––––– MISMATCH DETECTED –––––')
print(e)
except Exception as e:
print('––––– UNEXPECTED ERROR DURING COMPARISON –––––')
print(f'{type(e).__name__}: {e}')
if __name__ == '__main__':
main()
```
```python
## nas_model_2
@basic_unit
class ReciprocalWrapper(nni_nn.Module):
def forward(self, x):
return torch.reciprocal(x)
```
### Triage
* needs-triage
Contributor guide
No contributing guide indexed for this repository
Research direction
Run the supplied Python reproducer with the stated TVM and ONNXRuntime versions, starting at relay.frontend.from_onnx and relay.build. Isolate the InstanceNorm2d → Reciprocal → Conv2d sequence and compare intermediate outputs between the two runtimes. Done means the final outputs agree within rtol=0.01 and atol=0.001.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python, pytorch
- Domain
- compilers, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100