microsoft / microsoft/onnxruntime
[Build] Align CPython 3.14 CUDA 13 wheels with TensorRT 11
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
The official `onnxruntime-gpu==1.29.0` CPython 3.14 CUDA 13 wheels that include `TensorrtExecutionProvider` still link TensorRT 10 on both supported x64 platforms:
- Linux x86_64 requests `libnvinfer.so.10` and `libnvonnxparser.so.10`.
- Windows x64 requests `nvinfer_10.dll` and `nvonnxparser_10.dll`.
They therefore cannot load `TensorrtExecutionProvider` in a CPython 3.14 environment using the current TensorRT 11 CUDA 13 distribution, which supplies the TensorRT 11 ABI.
I propose moving the TensorRT-enabled CUDA 13 Python-wheel profile to CUDA 13.3 and TensorRT 11.2.1.2 coherently:
- Use CUDA 13.3 and cuDNN 9 for Windows x64 and Linux x86_64 wheels.
- Build the Windows x64 and Linux x86_64 TensorRT providers against TensorRT 11.2.1.2.
- Apply the version change to every Python ABI built by the CUDA 13 packaging pipeline, including CPython 3.14.
- Leave Linux aarch64, CUDA 12, and non-Python package pipelines unchanged. The current Linux aarch64 wheel does not contain a TensorRT provider, so it is not a target for this ABI fix.
NVIDIA publishes TensorRT 11.2.1.2 packages built against CUDA 13.3 for Linux x86_64 and Windows x64.
### Urgency
This blocks `TensorrtExecutionProvider` in CPython 3.14 environments using the current TensorRT 11 CUDA 13 distribution.
### Target platforms
- Linux x86_64: CPython 3.14 (`cp314-cp314` and `cp314-cp314t`)
- Windows x64: CPython 3.14 (`cp314-cp314`)
- CUDA 13.3, cuDNN 9, and TensorRT 11.2.1.2 on both platforms
### Build scripts
The official pipeline path is:
- `tools/ci_build/github/azure-pipelines/py-cuda13-packaging-pipeline.yml`
- `tools/ci_build/github/azure-pipelines/stages/py-gpu-packaging-stage.yml`
- `tools/ci_build/github/azure-pipelines/stages/py-linux-gpu-stage.yml`
- `tools/ci_build/github/azure-pipelines/stages/py-win-gpu-stage.yml`
- `tools/ci_build/github/azure-pipelines/templates/jobs/download_win_gpu_library.yml`
- `tools/ci_build/github/linux/build_linux_python_package.sh`
The current pipeline uses CUDA 13.0. Its x64 TensorRT inputs are TensorRT 10.x.
The CUDA 13 packaging pipeline therefore needs a CUDA 13.3 Linux x86_64 build image and matching Windows CUDA, cuDNN, and TensorRT inputs.
### Error / output
Linux wheel:
`onnxruntime_gpu-1.29.0-cp314-cp314-manylinux_2_28_x86_64.whl`
SHA-256:
`5c50d24541bf61e668dfe0a7a3a62de98def81959c832cc2d376814b661cb006`
Relevant `readelf -d libonnxruntime_providers_tensorrt.so` output:
```text
Shared library: [libcublas.so.13]
Shared library: [libnvinfer.so.10]
Shared library: [libnvonnxparser.so.10]
Shared library: [libcudart.so.13]
Shared library: [libcudnn.so.9]
```
Windows wheel:
`onnxruntime_gpu-1.29.0-cp314-cp314-win_amd64.whl`
SHA-256:
`4247025e674a666f839c02b90c829277cb2e1be11576d3646cad5c86b7fc26c5`
Relevant `objdump -p onnxruntime_providers_tensorrt.dll` output:
```text
DLL Name: cublas64_13.dll
DLL Name: nvinfer_10.dll
DLL Name: nvonnxparser_10.dll
DLL Name: cudnn64_9.dll
```
Expected TensorRT dependencies:
```text
Linux: libnvinfer.so.11, libnvonnxparser.so.11
Windows: nvinfer_11.dll, nvonnxparser_11.dll
```
The CPython 3.14 Linux aarch64 wheel was also inspected and contains CUDA and shared providers, but no TensorRT provider; it is therefore outside this issue's target platform set.
### GCC / Compiler Version
The current Linux x86_64 CUDA 13 packaging image uses GCC 14.
### References
- https://docs.nvidia.com/deeplearning/tensorrt/latest/getting-started/release-notes-11/11.2.1.html
- https://docs.nvidia.com/deeplearning/tensorrt/latest/installing-tensorrt/prerequisites.html
- https://github.com/NVIDIA/TensorRT/blob/main/README.md
- https://github.com/microsoft/onnxruntime/issues/26547
- https://github.com/microsoft/onnxruntime/pull/28611
- https://pypi.org/project/onnxruntime-gpu/1.29.0/
Contributor guide
Research direction
Start with tools/ci_build/github/azure-pipelines/py-cuda13-packaging-pipeline.yml and its GPU stage files, then trace the CUDA, cuDNN, and TensorRT inputs through download_win_gpu_library.yml and build_linux_python_package.sh. Verify the resulting Linux x86_64 and Windows x64 Python wheels request TensorRT 11 libraries while Linux aarch64, CUDA 12, and non-Python pipelines remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100