microsoft / microsoft/onnxruntime-genai

[GPU] Device selection issue

Open
#1,545 6 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

Hello,
I'm currently working based on the commit of [f0ae822], I'm using several GPUs on my server, so I need to select the device.
I have both explored C++ and python versions:
in C++, when I do:
Oga::SetCurrentGpuDeviceId(3);
then I got:

2025-06-11 21:13:01.366170206 [E:onnxruntime:, inference_session.cc:2117 operator()] Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:129 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, SUCCTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; SUCCTYPE = cudnnStatus_t; std::conditional_t<THRW, void, common::Status> = void] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:121 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, SUCCTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; SUCCTYPE = cudnnStatus_t; std::conditional_t<THRW, void, common::Status> = void] CUDNN failure 2007: CUDNN_STATUS_BAD_PARAM_STREAM_MISMATCH ; GPU=0 ; hostname=gpumixt ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=183 ; expr=cudnnSetStream(cudnn_handle_, stream);

terminate called after throwing an instance of 'std::runtime_error'
what(): Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:129 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, SUCCTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; SUCCTYPE = cudnnStatus_t; std::conditional_t<THRW, void, common::Status> = void] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:121 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, SUCCTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; SUCCTYPE = cudnnStatus_t; std::conditional_t<THRW, void, common::Status> = void] CUDNN failure 2007: CUDNN_STATUS_BAD_PARAM_STREAM_MISMATCH ; GPU=0 ; hostname=gpumixt ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=183 ; expr=cudnnSetStream(cudnn_handle_, stream);

Otherwise, if I use python:
import onnxruntime_genai_cuda as og
og.set_current_gpu_device_id(1)

I get the segmentation fault directly.

My code works very well, if I don't select the GPU number. But I need to absolutely use this function.

Anyone has any idea?
Thank you

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start from Oga::SetCurrentGpuDeviceId(3) and og.set_current_gpu_device_id(1), using commit f0ae822 and the reported CUDNN error in cuda_execution_provider.cc. Reproduce both the C++ initialization failure and Python segmentation fault, then trace GPU selection through initialization. Done means both bindings select the requested GPU without the stream-mismatch error or a crash.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
ai, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.