Docling-serve Fails to Utilize GPU Despite Available CUDA Provider

## Docling-serve Fails to Utilize GPU Despite Available CUDA Provider

**Environment:**

* **OS:** Windows with Docker Desktop (WSL 2 backend)
* **GPU:** NVIDIA GeForce RTX 5070
* **NVIDIA Driver:** 581.57
* **CUDA Version (reported by driver):** 13.0

**Goal:**

Run `docling-serve` within Docker, utilizing the NVIDIA GPU for acceleration (e.g., for OCR, VLM).

**Problem Description:**

Attempts to run `docling-serve` with GPU support have encountered several issues leading to the GPU not being utilized, even when the container environment seems correctly configured.

**Steps Taken & Issues Encountered:**

1. **Image `ghcr.io/docling-project/docling-serve:latest`:** Container runs, but uses CPU only, as expected.
2. **Image `ghcr.io/docling-project/docling-serve-cu126`:** Container fails during job processing with `torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device`. This is likely due to incompatibility between the image's CUDA 12.6 build and the host's CUDA 13.0 capability.
3. **Image `ghcr.io/docling-project/docling-serve-cu128`:**
* Container starts and `nvidia-smi` *inside* the container correctly identifies the GPU.
* However, `nvidia-smi` shows `N/A` for GPU Memory Usage for the main python process (PID 1), even during active jobs.
* Investigation revealed that this image incorrectly contains the `onnxruntime` (CPU) package instead of `onnxruntime-gpu`. The available providers list confirmed only `CPUExecutionProvider` was present.
4. **Custom Image Build (Workaround):**
* Created a `Dockerfile` based on `ghcr.io/docling-project/docling-serve-cu128`.
* Added steps to `USER root`, `pip uninstall -y onnxruntime`, `pip install --no-cache-dir onnxruntime-gpu`, and `USER 1001`.
* Built and ran this custom image using Docker Compose.
* **Verification inside the container (using `docker exec`):**
* `pip show onnxruntime onnxruntime-gpu` confirms `onnxruntime-gpu` is installed and `onnxruntime` is removed. ✅
* `python -c "import onnxruntime as ort; print(ort.get_available_providers())"` outputs `['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']`, confirming the CUDA provider is now available. ✅
* **Persistent Issue:** Despite the corrected setup and available CUDA provider, running `watch -n 1 nvidia-smi` inside the container during an active job *still* shows `N/A` GPU Memory Usage and 0% GPU-Util for the main python process (PID 1).

**Configuration Snippets:**

* **`docker-compose.yml` (relevant service):**
```yaml
docling:
build:
context: .
dockerfile: Dockerfile
container_name: docling
restart: unless-stopped
ports:
- "5011:5001"
environment:
- DOCLING_SERVE_ENABLE_UI=true
- DOCLING_SERVE_MAX_SYNC_WAIT=1200
- DOCLING_SERVE_ARTIFACTS_PATH=/app/models
- NVIDIA_VISIBLE_DEVICES=all
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
networks:
- caddy_network
volumes:
- doclingdata:/app/data
- doclingmodels:/app/models # Pre-populated using cp from base image cache
```
* **`Dockerfile`:**
```dockerfile
FROM ghcr.io/docling-project/docling-serve-cu128

USER root

RUN pip uninstall -y onnxruntime && \
pip install --no-cache-dir onnxruntime-gpu

USER 1001
```

**Conclusion:**

Even with a seemingly correct environment (GPU visible, correct `onnxruntime-gpu` installed, CUDA provider available), `docling-serve` appears to default to the CPU provider and does not utilize the GPU. This suggests an issue within Docling's logic for selecting the ONNX Runtime execution provider.