onnxruntime-gpu fails to find libnvrtc.so.12 when CUDA is not installed globally

Describe the issue

When using the onnxruntime-gpu wheel (installed via PyPI) in an environment without system-wide CUDA installation, loading a InferenceSession with providers=["CUDAExecutionProvider"] fails due to a missing libnvrtc.so.12:

2025-05-12 06:25:10.232591294 [E:onnxruntime:Default, provider_bridge_ort.cc:2195 TryGetProviderInfo_CUDA] 

/onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1778 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() 

[ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error:

 libnvrtc.so.12: cannot open shared object file: No such file or directory

I didn't have CUDA installed on the machine, so I use pypi-wheels

Importing torch before InferenceSession does not affect the outcome.(PyTorch Version: 2.5.1+cu124)

To make it work, I must explicitly call onnxruntime.preload_dlls() or manually load libnvrtc.so.12 using ctypes.CDLL before creating any InferenceSession(load_nvrtc).

To reproduce

import ctypes
import glob
import os
import sys
from pathlib import Path

import torch

pass

import onnxruntime


def load_nvrtc():
    import torch

    if not torch.cuda.is_available():
        print("[INFO] CUDA is not available, skipping nvrtc setup.")
        return

    if sys.platform == "win32":
        torch_lib_dir = Path(torch.__file__).parent / "lib"
        if torch_lib_dir.exists():
            os.add_dll_directory(str(torch_lib_dir))
            print(f"[INFO] Added DLL directory: {torch_lib_dir}")
            pattern = str(torch_lib_dir / "nvrtc*.dll")
            matching_files = sorted(glob.glob(pattern))
            if not matching_files:
                print(f"[ERROR] No nvrtc*.dll found in {torch_lib_dir}")
                return
            for dll_path in matching_files:
                dll_name = os.path.basename(dll_path)
                try:
                    ctypes.CDLL(dll_name)
                    print(f"[INFO] Loaded: {dll_name}")
                except OSError as e:
                    print(f"[WARNING] Failed to load {dll_name}: {e}")
        else:
            print(f"[WARNING] Torch lib directory not found: {torch_lib_dir}")

    elif sys.platform == "linux":
        site_packages = Path(torch.__file__).resolve().parents[1]
        nvrtc_dir = site_packages / "nvidia" / "cuda_nvrtc" / "lib"

        if not nvrtc_dir.exists():
            print(f"[ERROR] nvrtc dir not found: {nvrtc_dir}")
            return

        pattern = str(nvrtc_dir / "libnvrtc*.so*")
        matching_files = sorted(glob.glob(pattern))
        if not matching_files:
            print(f"[ERROR] No libnvrtc*.so* found in {nvrtc_dir}")
            return

        for so_path in matching_files:
            try:
                ctypes.CDLL(so_path, mode=ctypes.RTLD_GLOBAL)
                print(f"[INFO] Loaded: {so_path}")
            except OSError as e:
                print(f"[WARNING] Failed to load {so_path}: {e}")


# onnxruntime.preload_dlls()
# load_nvrtc()

sess_options = onnxruntime.SessionOptions()
sess_options.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
sess_options.execution_mode = onnxruntime.ExecutionMode.ORT_SEQUENTIAL
sess_options.intra_op_num_threads = 2

onnxruntime.set_default_logger_severity(3)
onnxruntime.InferenceSession(
    "/workspace/GPT-SoVITS/GPT_SoVITS/text/G2PWModel/g2pW.onnx",
    sess_options=sess_options,
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)

Urgency

Not Urgent

Platform

Linux

OS Version

Ubuntu 22.04

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.22.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 12.4 From Pypi Wheels