microsoft / microsoft/onnxruntime
One ONNX graph will result in an "UpdateState bias should be 1D" Error on GPU but success on CPU
@RandySheriffH is already working on this.
Since Apr 21, 2022.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 8h
- Merged PRs (30d)
- 179
Description
Describe the bug
The graph that result in this issue (can be accessed through this link):

When I run this graph on CPU mode, it works normally with no error, but when I run this graph on GPU mode, it throws a runtime error as follows:
[/usr/local/lib/python3.7/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py](https://localhost:8080/#) in run(self, output_names, input_feed, run_options)
190 output_names = [output.name for output in self._outputs_meta]
191 try:
--> 192 return self._sess.run(output_names, input_feed, run_options)
193 except C.EPFail as err:
194 if self._enable_fallback:
Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running FusedConv node. Name:'model_44/conv2d_37/Conv2D' Status Message: conv.cc:223 UpdateState bias should be 1D
To Reproduce
- Describe steps/code to reproduce the behavior.
- Attach the ONNX model to the issue (where applicable) to expedite investigation.
Please first download the onnx graph through: https://drive.google.com/file/d/1mHUJuhjUxGjjWmpwwXe8NS-jOXhEts4y/view?usp=sharing
Then run the following code:
import onnxruntime as ort
providers = [
('CUDAExecutionProvider', {
'device_id': 0,
'arena_extend_strategy': 'kNextPowerOfTwo',
'gpu_mem_limit': 10 * 1024 * 1024 * 1024, # 10G
'cudnn_conv_algo_search': 'EXHAUSTIVE',
'do_copy_in_default_stream': True,
}),
'CPUExecutionProvider',
]
onnx_path = "bug.onnx"
model = ort.InferenceSession(onnx_path, providers=providers)
import numpy as np
input = np.random.rand(10, 36, 36, 528)
input_name = model.get_inputs()[0].name
output_name = model.get_outputs()[0].name
input = input.astype('float32')
pred = model.run([output_name], {input_name: input})[0]
print("When running on GPU: ", pred[0])
model.set_providers(['CPUExecutionProvider'])
input_name = model.get_inputs()[0].name
output_name = model.get_outputs()[0].name
input = input.astype('float32')
pred = model.run([output_name], {input_name: input})[0]
print("When running on CPU: ", pred[0])
System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
- ONNX Runtime installed from (source or binary): binary
- ONNX Runtime version: latest
- Python version: 3.7
- Visual Studio version (if applicable): N/A
- GCC/Compiler version (if compiling from source): N/A
- CUDA/cuDNN version: 11.2
- GPU model and memory: Tesla K80 (11441MiB)
Expected behavior
ONNXRuntime should behave the same between GPU context and CPU context. And in this case, I think it should work normally in GPU mode.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.