microsoft / microsoft/onnxruntime

[Build] Align CPython 3.14 CUDA 13 wheels with TensorRT 11

Open
#32,278 1 comment 0 reactions 0 assignees View on GitHub
ep:CUDA ep:TensorRT
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

The official `onnxruntime-gpu==1.29.0` CPython 3.14 CUDA 13 wheels that include `TensorrtExecutionProvider` still link TensorRT 10 on both supported x64 platforms:

- Linux x86_64 requests `libnvinfer.so.10` and `libnvonnxparser.so.10`.
- Windows x64 requests `nvinfer_10.dll` and `nvonnxparser_10.dll`.

They therefore cannot load `TensorrtExecutionProvider` in a CPython 3.14 environment using the current TensorRT 11 CUDA 13 distribution, which supplies the TensorRT 11 ABI.

I propose moving the TensorRT-enabled CUDA 13 Python-wheel profile to CUDA 13.3 and TensorRT 11.2.1.2 coherently:

- Use CUDA 13.3 and cuDNN 9 for Windows x64 and Linux x86_64 wheels.
- Build the Windows x64 and Linux x86_64 TensorRT providers against TensorRT 11.2.1.2.
- Apply the version change to every Python ABI built by the CUDA 13 packaging pipeline, including CPython 3.14.
- Leave Linux aarch64, CUDA 12, and non-Python package pipelines unchanged. The current Linux aarch64 wheel does not contain a TensorRT provider, so it is not a target for this ABI fix.

NVIDIA publishes TensorRT 11.2.1.2 packages built against CUDA 13.3 for Linux x86_64 and Windows x64.

### Urgency

This blocks `TensorrtExecutionProvider` in CPython 3.14 environments using the current TensorRT 11 CUDA 13 distribution.

### Target platforms

- Linux x86_64: CPython 3.14 (`cp314-cp314` and `cp314-cp314t`)
- Windows x64: CPython 3.14 (`cp314-cp314`)
- CUDA 13.3, cuDNN 9, and TensorRT 11.2.1.2 on both platforms

### Build scripts

The official pipeline path is:

- `tools/ci_build/github/azure-pipelines/py-cuda13-packaging-pipeline.yml`
- `tools/ci_build/github/azure-pipelines/stages/py-gpu-packaging-stage.yml`
- `tools/ci_build/github/azure-pipelines/stages/py-linux-gpu-stage.yml`
- `tools/ci_build/github/azure-pipelines/stages/py-win-gpu-stage.yml`
- `tools/ci_build/github/azure-pipelines/templates/jobs/download_win_gpu_library.yml`
- `tools/ci_build/github/linux/build_linux_python_package.sh`

The current pipeline uses CUDA 13.0. Its x64 TensorRT inputs are TensorRT 10.x.

The CUDA 13 packaging pipeline therefore needs a CUDA 13.3 Linux x86_64 build image and matching Windows CUDA, cuDNN, and TensorRT inputs.

### Error / output

Linux wheel:

`onnxruntime_gpu-1.29.0-cp314-cp314-manylinux_2_28_x86_64.whl`

SHA-256:

`5c50d24541bf61e668dfe0a7a3a62de98def81959c832cc2d376814b661cb006`

Relevant `readelf -d libonnxruntime_providers_tensorrt.so` output:

```text
Shared library: [libcublas.so.13]
Shared library: [libnvinfer.so.10]
Shared library: [libnvonnxparser.so.10]
Shared library: [libcudart.so.13]
Shared library: [libcudnn.so.9]
```

Windows wheel:

`onnxruntime_gpu-1.29.0-cp314-cp314-win_amd64.whl`

SHA-256:

`4247025e674a666f839c02b90c829277cb2e1be11576d3646cad5c86b7fc26c5`

Relevant `objdump -p onnxruntime_providers_tensorrt.dll` output:

```text
DLL Name: cublas64_13.dll
DLL Name: nvinfer_10.dll
DLL Name: nvonnxparser_10.dll
DLL Name: cudnn64_9.dll
```

Expected TensorRT dependencies:

```text
Linux: libnvinfer.so.11, libnvonnxparser.so.11
Windows: nvinfer_11.dll, nvonnxparser_11.dll
```

The CPython 3.14 Linux aarch64 wheel was also inspected and contains CUDA and shared providers, but no TensorRT provider; it is therefore outside this issue's target platform set.

### GCC / Compiler Version

The current Linux x86_64 CUDA 13 packaging image uses GCC 14.

### References

- https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/release-notes-11/11.2.1.html
- https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/prerequisites.html
- https://github.com/NVIDIA/TensorRT/blob/main/README.md
- https://github.com/microsoft/onnxruntime/issues/26547
- https://github.com/microsoft/onnxruntime/pull/28611
- https://pypi.org/project/onnxruntime-gpu/1.29.0/

Contributor guide

Open the contributing guide

Research direction

Start with tools/ci_build/github/azure-pipelines/py-cuda13-packaging-pipeline.yml and its GPU stage files, then trace the CUDA, cuDNN, and TensorRT inputs through download_win_gpu_library.yml and build_linux_python_package.sh. Verify the resulting Linux x86_64 and Windows x64 Python wheels request TensorRT 11 libraries while Linux aarch64, CUDA 12, and non-Python pipelines remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.