microsoft / microsoft/onnxruntime

[Performance] RE: When using CUDA the first run is very slow -- cudnn_conv_algo_search

Open
#19,838 7 comments 5 reactions 0 assignees View on GitHub
documentation
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

I didn't want to reply to https://github.com/microsoft/onnxruntime/issues/10746 since it was mentionned that the issue is a placeholder.

I wanted to say that in our work, we've found that issue to have omitted a critical piece of information regarding the effect of `cudnn_conv_algo_search` to the performance of the first run.

The default value, `EXAUSTIVE` as mentioned in the C API and the Python documentation

* https://onnxruntime.ai/docs/api/c/struct_ort_c_u_d_a_provider_options.html#acc33f79bc3024bfadf2950d139d56370
* https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#cudnn_conv_algo_search

Seems to be a significant contributor to this effect.
It would be good if a small note were added in that placeholder issue to mention that users would have a choice in the session optimization strategy.

Thank you @davidmezzetti for bringing thing to my attention in your blog post
https://medium.com/neuml/debug-onnx-gpu-performance-c9290fe07459

cc: @jefromson

### To reproduce

Start your onnx session with the following options:

and change between the different options for `cudnn_conv_algo_search`
```
providers=[
("CUDAExecutionProvider", {
# "cudnn_conv_algo_search": "DEFAULT",
# "cudnn_conv_algo_search": "HEURISTIC",
"cudnn_conv_algo_search": "EXHAUSTIVE",
}),
# "CPUExecutionProvider",
]
```

### Urgency

just a small tip for others.

### Platform

Linux

### OS Version

Ubuntu 22.04

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.17.1

### ONNX Runtime API

Python

### Architecture

X64

### Execution Provider

CUDA

### Execution Provider Library Version

CUDA 12.0

### Model File

_No response_

### Is this a quantized model?

No

Contributor guide

Open the contributing guide

Research direction

Start with the linked placeholder issue 10746 and the CUDA Execution Provider documentation sections for cudnn_conv_algo_search. Confirm how DEFAULT, HEURISTIC, and EXHAUSTIVE affect first-run performance, then add a brief note about the available session optimization choices to the placeholder issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, machine-learning, performance
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.