microsoft / microsoft/onnxruntime-genai

Setting specific device_id with set_current_gpu_device_id not working

Open
#730 2 comments 0 reactions 0 assignees View on GitHub
bug ep:CUDA
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

**Describe the bug**
Hello. I tried to use a method to set the device_id, even if not documented. Sadly it works only on device 0

**To Reproduce**
This works
```python
import onnxruntime_genai as og

# set device id
og.set_current_gpu_device_id(0)
```

This doesn't
```python
import onnxruntime_genai as og

# set device id
og.set_current_gpu_device_id(1)
```

I get this error

```shell
2024-07-29 15:55:02.094196786 [E:onnxruntime:onnxruntime-genai, inference_session.cc:2045 operator()] Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:123 std::conditional_t onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; std::conditional_t = void] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:116 std::conditional_t onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; std::conditional_t = void] CUDNN failure 7: CUDNN_STATUS_MAPPING_ERROR ; GPU=0 ; hostname=041ac24945f2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=182 ; expr=cudnnSetStream(cudnn_handle_, stream);

Traceback (most recent call last):
File "", line 1, in
onnxruntime_genai.onnxruntime_genai.OrtException: Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:123 std::conditional_t onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; std::conditional_t = void] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:116 std::conditional_t onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; std::conditional_t = void] CUDNN failure 7: CUDNN_STATUS_MAPPING_ERROR ; GPU=0 ; hostname=041ac24945f2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=182 ; expr=cudnnSetStream(cudnn_handle_, stream);
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the Python entry point set_current_gpu_device_id and trace how the selected device reaches inference initialization. Reproduce the two snippets with device IDs 0 and 1, then inspect the CUDNN_STATUS_MAPPING_ERROR showing GPU=0. Done means selecting device 1 initializes successfully instead of failing while reporting GPU=0.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.