docling-project / docling-project/docling
RapidOCR on CUDA is 4.4x slower than CPU because cudnn_conv_algo_search defaults to EXHAUSTIVE
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Bug
#4103 made RapidOCR's ONNX Runtime backend use the CUDA execution provider, and
`docs/usage/gpu.md` now documents how to enable it. On this machine the result is that
OCR gets **4.4x slower than the CPU it replaces**.
RapidOCR defaults `EngineConfig.onnxruntime.cuda_ep_cfg.cudnn_conv_algo_search` to
`EXHAUSTIVE` (`rapidocr/config.yaml`), which re-searches the convolution algorithms every
time an input shape changes. Detection sees one shape per page size and amortizes that
search. Recognition feeds one text-line crop at a time, so its shape changes on nearly
every call and the search never pays for itself — it costs far more than the inference.
Measured on `1c2b794b` with nothing patched, an L4 with onnxruntime-gpu 1.29.0 and
rapidocr 3.9.2, one 1224x1584 page with `lang=["japan"]` (35 text lines), median of three
runs after a warm-up (spread within 0.01s):
| | det | cls | rec | total |
| -------------------------- | ----- | ----- | ------ | ------ |
| CPU | 0.96s | 0.03s | 1.66s | 2.65s |
| CUDA (main today) | 0.12s | 0.21s | 11.26s | 11.60s |
| CUDA + `DEFAULT` search | 0.22s | 0.16s | 1.30s | 1.68s |
Detection does get 8x faster on the GPU, as expected. Recognition is what inverts the
result.
Passing `"EngineConfig.onnxruntime.cuda_ep_cfg.cudnn_conv_algo_search": "DEFAULT"`
alongside the keys #4103 added turns the GPU path into a 1.6x speedup over CPU instead of
a 4.4x slowdown. `HEURISTIC` does not help — it still searches (11.72s); only `DEFAULT`
skips it.
The trade-off is small and one-sided: `DEFAULT` costs detection 0.08s a page, because a
fixed algorithm is not the one `EXHAUSTIVE` would have found, while recognition saves
about 10s. Setting it per model would be better still, but RapidOCR binds one engine
config to all three (`cfg.Det.engine_cfg = cfg.EngineConfig[engine_type]`), so it cannot
be chosen per model from Docling.
For the record, this is not a per-node fallback to the CPU. With CUDA requested,
`session_state` reports `All nodes placed on [CUDAExecutionProvider]. Number of nodes:
190` for the detection model, with `MemcpyTransformer modified: 0`.
### Steps to reproduce
On a CUDA host with `onnxruntime-gpu` installed, following the setup in
`docs/usage/gpu.md`:
```python
from docling.datamodel.accelerator_options import AcceleratorOptions
from docling.datamodel.pipeline_options import RapidOcrOptions
from docling.models.stages.ocr.rapid_ocr_model import RapidOcrModel
model = RapidOcrModel(
enabled=True,
artifacts_path=None,
options=RapidOcrOptions(backend="onnxruntime", lang=["japan"]),
accelerator_options=AcceleratorOptions(device="cuda", num_threads=4),
)
# time model.reader(page_image_array) against the same call with device="cpu"
```
### Docling version
2.125.0 (measured on `1c2b794b`)
### Python version
3.13.15
Contributor guide
Research direction
Start with rapidocr/config.yaml and the RapidOcrModel entry point, then trace how EngineConfig.onnxruntime.cuda_ep_cfg is passed to the ONNX Runtime CUDA provider. Use the setup in docs/usage/gpu.md and the supplied reproduction to compare CPU and CUDA recognition timings; done means the CUDA path no longer incurs the reported repeated convolution-search slowdown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, performance
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100