microsoft / microsoft/onnxruntime

One ONNX graph will result in an "UpdateState bias should be 1D" Error on GPU but success on CPU

Open
#11,241 5 comments 4 reactions 1 assignee View on GitHub

@RandySheriffH is already working on this.

Since Apr 21, 2022.

core runtime ep:CUDA
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 8h
Merged PRs (30d)
179

Description

Describe the bug
The graph that result in this issue (can be accessed through this link):
image

When I run this graph on CPU mode, it works normally with no error, but when I run this graph on GPU mode, it throws a runtime error as follows:

[/usr/local/lib/python3.7/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py](https://localhost:8080/#) in run(self, output_names, input_feed, run_options)
    190             output_names = [output.name for output in self._outputs_meta]
    191         try:
--> 192             return self._sess.run(output_names, input_feed, run_options)
    193         except C.EPFail as err:
    194             if self._enable_fallback:

Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running FusedConv node. Name:'model_44/conv2d_37/Conv2D' Status Message: conv.cc:223 UpdateState bias should be 1D

To Reproduce

  • Describe steps/code to reproduce the behavior.
  • Attach the ONNX model to the issue (where applicable) to expedite investigation.

Please first download the onnx graph through: https://drive.google.com/file/d/1mHUJuhjUxGjjWmpwwXe8NS-jOXhEts4y/view?usp=sharing
Then run the following code:

import onnxruntime as ort
providers = [
    ('CUDAExecutionProvider', {
        'device_id': 0,
        'arena_extend_strategy': 'kNextPowerOfTwo',
        'gpu_mem_limit': 10 * 1024 * 1024 * 1024,  # 10G
        'cudnn_conv_algo_search': 'EXHAUSTIVE',
        'do_copy_in_default_stream': True,
    }),
    'CPUExecutionProvider',
]
onnx_path = "bug.onnx"
model = ort.InferenceSession(onnx_path, providers=providers)

import numpy as np
input = np.random.rand(10, 36, 36, 528)

input_name = model.get_inputs()[0].name
output_name = model.get_outputs()[0].name
input = input.astype('float32')
pred = model.run([output_name], {input_name: input})[0]
print("When running on GPU: ", pred[0])

model.set_providers(['CPUExecutionProvider'])
input_name = model.get_inputs()[0].name
output_name = model.get_outputs()[0].name
input = input.astype('float32')
pred = model.run([output_name], {input_name: input})[0]
print("When running on CPU: ", pred[0])

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
  • ONNX Runtime installed from (source or binary): binary
  • ONNX Runtime version: latest
  • Python version: 3.7
  • Visual Studio version (if applicable): N/A
  • GCC/Compiler version (if compiling from source): N/A
  • CUDA/cuDNN version: 11.2
  • GPU model and memory: Tesla K80 (11441MiB)

Expected behavior
ONNXRuntime should behave the same between GPU context and CPU context. And in this case, I think it should work normally in GPU mode.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.