ONNXRuntime-GPU Fails in ComfyUI – Possible cuDNN Conflict with CTranslate2

Environment

• platform: Linux-6.8.0-53-generic-x86_64-with-glibc2.39
• CUDA: 12.4
• Driver: 550.144.03
• Python 3.12.3

Key Packages

• torch 2.6.0+cu124
• ctranslate2 4.5.0
• onnxruntime-gpu 1.22.0
• nvidia-cublas-cu12 12.4.5.8
• nvidia-cuda-cupti-cu12 12.4.127
• nvidia-cuda-nvrtc-cu12 12.4.127
• nvidia-cuda-runtime-cu12 12.4.127
• nvidia-cudnn-cu12 9.1.0.70
• nvidia-cufft-cu12 11.2.1.3
• nvidia-curand-cu12 10.3.5.147
• nvidia-cusolver-cu12 11.6.1.9
• nvidia-cusparse-cu12 12.3.1.170
• nvidia-cusparselt-cu12 0.6.2
• nvidia-ml-py 12.570.86
• nvidia-nccl-cu12 2.21.5
• nvidia-nvjitlink-cu12 12.4.127
• nvidia-nvtx-cu12 12.4.127

Error (When running ComfyUI with ONNXRuntime-GPU)

[E:onnxruntime:, inference_session.cc:2105 operator()] Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:123 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; std::conditional_t<THRW, void, common::Status> = void] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:116 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; std::conditional_t<THRW, void, common::Status> = void] CUDNN failure 4: cuDNN could not be found or could not be loaded. ; GPU=0 ; hostname=gpu8 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=181 ; expr=cudnnCreate(&cudnn_handle_);
!!! Exception during processing !!! [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:123 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; std::conditional_t<THRW, void, common::Status> = void] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:116 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; std::conditional_t<THRW, void, common::Status> = void] CUDNN failure 4: cuDNN could not be found or could not be loaded. ; GPU=0 ; hostname=gpu8 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=181 ; expr=cudnnCreate(&cudnn_handle_);

Suspected Cause

In the same Python environment, this error does not occur when running code directly — it only happens inside ComfyUI.

ComfyUI loads cuDNN from CTranslate2 (libcudnn-74a4c495.so.9.1.0) first, which may conflict with ONNXRuntime-GPU.
When I use:

export LD_PRELOAD=env/lib/python3.12/site-packages/nvidia/cudnn/lib/libcudnn.so.9

The error disappears.

Question

Is there a better way to avoid this cuDNN conflict without using LD_PRELOAD every time?
Any clean solution to make ONNXRuntime-GPU and CTranslate2 coexist in the same Python environment?

No project files or tests are named. Start by reproducing the ComfyUI failure and comparing library loading with direct Python execution, focusing on CTranslate2's cuDNN loading and ONNXRuntime-GPU initialization. Done means both packages coexist in one environment without requiring LD_PRELOAD.