microsoft / microsoft/onnxruntime

GPT-Neo: Torch CUDA 2x faster than ONNX CUDA

Open
#7,238 16 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ep:CUDA ep:TensorRT feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 8h
Merged PRs (30d)
179

Description

Describe the bug
I have been heavily investigating GPT-Neo for our company. Most of our models run directly on GPU with ONNX as backend. The problem is as follows:

Running the gpt-neo-1.3B model in a custom build onnxruntime instance on a V100 is 2x slower than base pytorch (onnx: 300ms vs torch: 120 ms). This is usually exactly the other way around.

I would assume that this is either a problem of gpt-neo-1.3B or of onnxruntime, most probably the first one. But I am now sure how to best determine the responsible part in this chain and how we can go ahead without altering the model significantly and manually.

Help would be very appreciated.

Urgency
None

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
  • ONNX Runtime installed from (source or binary): Source
  • ONNX Runtime version: 1.7.1
  • Python version: 3. 8
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version: 11.1.1 / 8.0.5
  • GPU model and memory: Nvidia V100

To Reproduce

  1. Build the following Dockerfile (may take some time)

https://github.com/oborchers/Medium_Repo/blob/master/onnxruntime-issues/Dockerfile

  1. Run the following notebook:

https://github.com/oborchers/Medium_Repo/blob/master/onnxruntime-issues/GPT-Neo%20Slow-2.ipynb

Torch: 1000 sentences took 53s at 18 sentences/s.
ONNX: 1000 sentences took 313s at 3 sentences/s.
Expected behavior
I would expect onnxruntime inference to be at least as fast as torch, which really surprised me.

Partial Log:

2021-04-05 16:09:30.169135381 [W:onnxruntime:Default, cuda_execution_provider.cc:1983 GetCapability] CUDA kernel not found in registries for Op type: Pad node name: Pad_4368
2021-04-05 16:09:30.169205679 [W:onnxruntime:Default, cuda_execution_provider.cc:1983 GetCapability] CUDA kernel not found in registries for Op type: Pad node name: Pad_4347
2021-04-05 16:09:30.169232416 [W:onnxruntime:Default, cuda_execution_provider.cc:1983 GetCapability] CUDA kernel not found in registries for Op type: Pad node name: Pad_4326
2021-04-05 16:09:30.169277052 [W:onnxruntime:Default, cuda_execution_provider.cc:1983 GetCapability] CUDA kernel not found in registries for Op type: Pad node name: Pad_4305
2021-04-05 16:09:30.169305005 [W:onnxruntime:Default, cuda_execution_provider.cc:1983 GetCapability] CUDA kernel not found in registries for Op type: Pad node name: Pad_4284
...
2021-04-05 16:09:30.171179525 [W:onnxruntime:Default, cuda_execution_provider.cc:1983 GetCapability] CUDA kernel not found in registries for Op type: Pad node name: Pad_3045
2021-04-05 16:09:30.171207277 [W:onnxruntime:Default, cuda_execution_provider.cc:1983 GetCapability] CUDA kernel not found in registries for Op type: LessOrEqual node name: LessOrEqual_7094
...
2021-04-05 16:09:30.358181987 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Gather_5
2021-04-05 16:09:30.358218137 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_17
2021-04-05 16:09:30.358230511 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Concat_18
2021-04-05 16:09:30.358241440 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Add_13
2021-04-05 16:09:30.358265649 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Slice_37
2021-04-05 16:09:30.358279929 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Squeeze_38
2021-04-05 16:09:30.358290325 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_85672
2021-04-05 16:09:30.358301675 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Gather_2
2021-04-05 16:09:30.358311978 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_85670
2021-04-05 16:09:30.358323299 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Concat_85673
2021-04-05 16:09:30.358344992 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Slice_85
2021-04-05 16:09:30.358356437 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Squeeze_86
2021-04-05 16:09:30.358373163 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Div_88
2021-04-05 16:09:30.358386774 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_93
2021-04-05 16:09:30.358399847 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Gather_80
2021-04-05 16:09:30.358410259 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_92
2021-04-05 16:09:30.358425290 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Gather_77
2021-04-05 16:09:30.358437256 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_91
2021-04-05 16:09:30.358452374 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Concat_94
2021-04-05 16:09:30.358476066 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Slice_63
2021-04-05 16:09:30.358490967 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Squeeze_64
2021-04-05 16:09:30.358503415 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Div_66
2021-04-05 16:09:30.358515382 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_71
2021-04-05 16:09:30.358528613 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Gather_58
2021-04-05 16:09:30.358538065 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_70
2021-04-05 16:09:30.358551073 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Gather_55
2021-04-05 16:09:30.358562240 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_69
2021-04-05 16:09:30.358578571 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Concat_72
2021-04-05 16:09:30.358601242 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Slice_132
2021-04-05 16:09:30.358612608 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Squeeze_133
2021-04-05 16:09:30.358623085 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_139
2021-04-05 16:09:30.358639190 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Slice_126
2021-04-05 16:09:30.358648692 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Squeeze_127
2021-04-05 16:09:30.358659862 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Sub_134
2021-04-05 16:09:30.358669326 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_135
2021-04-05 16:09:30.358690866 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Gather_11
2021-04-05 16:09:30.358703730 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_20
....
2021-04-05 17:37:21.034558149 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Gather_5
2021-04-05 17:37:21.034596725 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_17
2021-04-05 17:37:21.034649407 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Concat_18
2021-04-05 17:37:21.034679883 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Add_13
2021-04-05 17:37:21.034709999 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Slice_37
2021-04-05 17:37:21.034741824 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Squeeze_38
2021-04-05 17:37:21.034771635 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_85672
2021-04-05 17:37:21.034783311 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Gather_2
2021-04-05 17:37:21.034813241 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_85670
2021-04-05 17:37:21.034845762 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Concat_85673
2021-04-05 17:37:21.034885673 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Slice_85
2021-04-05 17:37:21.034918686 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Squeeze_86
2021-04-05 17:37:21.034930886 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Div_88
2021-04-05 17:37:21.035028500 [W:onnxruntime:Default, fallback_cpu_capability.cc:135 GetCpuPreferredNodes] Force fallback to CPU execution for node: Unsqueeze_93
...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by building the linked Dockerfile and running the linked GPT-Neo Slow-2.ipynb to reproduce the timing difference. Inspect the reported missing CUDA kernels and CPU fallback messages, then compare the model and ONNX Runtime execution paths; done requires identifying the responsible component and a confirmed path toward resolving the slowdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python, pytorch
Domain
machine-learning, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.