microsoft / microsoft/onnxruntime-genai
Setting specific device_id with set_current_gpu_device_id not working
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
**Describe the bug**
Hello. I tried to use a method to set the device_id, even if not documented. Sadly it works only on device 0
**To Reproduce**
This works
```python
import onnxruntime_genai as og
# set device id
og.set_current_gpu_device_id(0)
```
This doesn't
```python
import onnxruntime_genai as og
# set device id
og.set_current_gpu_device_id(1)
```
I get this error
```shell
2024-07-29 15:55:02.094196786 [E:onnxruntime:onnxruntime-genai, inference_session.cc:2045 operator()] Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:123 std::conditional_t onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; std::conditional_t = void] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:116 std::conditional_t onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; std::conditional_t = void] CUDNN failure 7: CUDNN_STATUS_MAPPING_ERROR ; GPU=0 ; hostname=041ac24945f2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=182 ; expr=cudnnSetStream(cudnn_handle_, stream);
Traceback (most recent call last):
File "", line 1, in
onnxruntime_genai.onnxruntime_genai.OrtException: Exception during initialization: /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:123 std::conditional_t onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; std::conditional_t = void] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:116 std::conditional_t onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudnnStatus_t; bool THRW = true; std::conditional_t = void] CUDNN failure 7: CUDNN_STATUS_MAPPING_ERROR ; GPU=0 ; hostname=041ac24945f2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=182 ; expr=cudnnSetStream(cudnn_handle_, stream);
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the Python entry point set_current_gpu_device_id and trace how the selected device reaches inference initialization. Reproduce the two snippets with device IDs 0 and 1, then inspect the CUDNN_STATUS_MAPPING_ERROR showing GPU=0. Done means selecting device 1 initializes successfully instead of failing while reporting GPU=0.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100