docling-project / docling-project/docling

RapidOCR on CUDA is 4.4x slower than CPU because cudnn_conv_algo_search defaults to EXHAUSTIVE

Open Beginner friendly
#4,167 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Bug

#4103 made RapidOCR's ONNX Runtime backend use the CUDA execution provider, and
`docs/usage/gpu.md` now documents how to enable it. On this machine the result is that
OCR gets **4.4x slower than the CPU it replaces**.

RapidOCR defaults `EngineConfig.onnxruntime.cuda_ep_cfg.cudnn_conv_algo_search` to
`EXHAUSTIVE` (`rapidocr/config.yaml`), which re-searches the convolution algorithms every
time an input shape changes. Detection sees one shape per page size and amortizes that
search. Recognition feeds one text-line crop at a time, so its shape changes on nearly
every call and the search never pays for itself — it costs far more than the inference.

Measured on `1c2b794b` with nothing patched, an L4 with onnxruntime-gpu 1.29.0 and
rapidocr 3.9.2, one 1224x1584 page with `lang=["japan"]` (35 text lines), median of three
runs after a warm-up (spread within 0.01s):

| | det | cls | rec | total |
| -------------------------- | ----- | ----- | ------ | ------ |
| CPU | 0.96s | 0.03s | 1.66s | 2.65s |
| CUDA (main today) | 0.12s | 0.21s | 11.26s | 11.60s |
| CUDA + `DEFAULT` search | 0.22s | 0.16s | 1.30s | 1.68s |

Detection does get 8x faster on the GPU, as expected. Recognition is what inverts the
result.

Passing `"EngineConfig.onnxruntime.cuda_ep_cfg.cudnn_conv_algo_search": "DEFAULT"`
alongside the keys #4103 added turns the GPU path into a 1.6x speedup over CPU instead of
a 4.4x slowdown. `HEURISTIC` does not help — it still searches (11.72s); only `DEFAULT`
skips it.

The trade-off is small and one-sided: `DEFAULT` costs detection 0.08s a page, because a
fixed algorithm is not the one `EXHAUSTIVE` would have found, while recognition saves
about 10s. Setting it per model would be better still, but RapidOCR binds one engine
config to all three (`cfg.Det.engine_cfg = cfg.EngineConfig[engine_type]`), so it cannot
be chosen per model from Docling.

For the record, this is not a per-node fallback to the CPU. With CUDA requested,
`session_state` reports `All nodes placed on [CUDAExecutionProvider]. Number of nodes:
190` for the detection model, with `MemcpyTransformer modified: 0`.

### Steps to reproduce

On a CUDA host with `onnxruntime-gpu` installed, following the setup in
`docs/usage/gpu.md`:

```python
from docling.datamodel.accelerator_options import AcceleratorOptions
from docling.datamodel.pipeline_options import RapidOcrOptions
from docling.models.stages.ocr.rapid_ocr_model import RapidOcrModel

model = RapidOcrModel(
enabled=True,
artifacts_path=None,
options=RapidOcrOptions(backend="onnxruntime", lang=["japan"]),
accelerator_options=AcceleratorOptions(device="cuda", num_threads=4),
)
# time model.reader(page_image_array) against the same call with device="cpu"
```

### Docling version

2.125.0 (measured on `1c2b794b`)

### Python version

3.13.15

Contributor guide

Open the contributing guide

Research direction

Start with rapidocr/config.yaml and the RapidOcrModel entry point, then trace how EngineConfig.onnxruntime.cuda_ep_cfg is passed to the ONNX Runtime CUDA provider. Use the setup in docs/usage/gpu.md and the supplied reproduction to compare CPU and CUDA recognition timings; done means the CUDA path no longer incurs the reported repeated convolution-search slowdown.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, performance
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.