microsoft / microsoft/onnxruntime
[Performance] RE: When using CUDA the first run is very slow -- cudnn_conv_algo_search
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
I didn't want to reply to https://github.com/microsoft/onnxruntime/issues/10746 since it was mentionned that the issue is a placeholder.
I wanted to say that in our work, we've found that issue to have omitted a critical piece of information regarding the effect of `cudnn_conv_algo_search` to the performance of the first run.
The default value, `EXAUSTIVE` as mentioned in the C API and the Python documentation
* https://onnxruntime.ai/docs/api/c/struct_ort_c_u_d_a_provider_options.html#acc33f79bc3024bfadf2950d139d56370
* https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#cudnn_conv_algo_search
Seems to be a significant contributor to this effect.
It would be good if a small note were added in that placeholder issue to mention that users would have a choice in the session optimization strategy.
Thank you @davidmezzetti for bringing thing to my attention in your blog post
https://medium.com/neuml/debug-onnx-gpu-performance-c9290fe07459
cc: @jefromson
### To reproduce
Start your onnx session with the following options:
and change between the different options for `cudnn_conv_algo_search`
```
providers=[
("CUDAExecutionProvider", {
# "cudnn_conv_algo_search": "DEFAULT",
# "cudnn_conv_algo_search": "HEURISTIC",
"cudnn_conv_algo_search": "EXHAUSTIVE",
}),
# "CPUExecutionProvider",
]
```
### Urgency
just a small tip for others.
### Platform
Linux
### OS Version
Ubuntu 22.04
### ONNX Runtime Installation
Released Package
### ONNX Runtime Version or Commit ID
1.17.1
### ONNX Runtime API
Python
### Architecture
X64
### Execution Provider
CUDA
### Execution Provider Library Version
CUDA 12.0
### Model File
_No response_
### Is this a quantized model?
No
Contributor guide
Research direction
Start with the linked placeholder issue 10746 and the CUDA Execution Provider documentation sections for cudnn_conv_algo_search. Confirm how DEFAULT, HEURISTIC, and EXHAUSTIVE affect first-run performance, then add a brief note about the available session optimization choices to the placeholder issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, machine-learning, performance
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100