microsoft / microsoft/onnxruntime-genai
Querying the ids of all the DML devices and selecting a device for inference
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 354
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 85
Description
I am working with `onnxruntime-genai-directml 0.2.0rc3` and Mistral-7B for LLM inference. The code in [`model-qa.py`](https://github.com/microsoft/onnxruntime-genai/blob/514493be37eebbfcdf387c3d83ac71650303d18a/examples/python/model-qa.py) works well for me in general. I tried the code on two different hardware:
1. A laptop with an Intel iGPU and Nvidia Laptop GPU
2. A desktop with an AMD iGPU and Nvidia Desktop GPU
The code works well on my laptop and it seems to use the Nvidia Laptop GPU for inference. However, on the desktop, the code tries using the AMD iGPU and crashes with an AMD driver timeout error. I disabled the iGPU on my desktop and ran the code again. It used the Nvidia Desktop GPU and worked well.
Apparently, the device with id 0 on my laptop is my Nvidia Laptop GPU whereas on my desktop it is the AMD iGPU (?) The Windows Task manager on my Laptop shows as follows:
Laptop:
```
GPU 0
Intel UHD Graphics
Physical location: PCI bus 0, device 2, function 0
GPU 1
Nvidia GeForce RTX... Laptop GPU
Physical location: PCI bus 1, device 0, function 0
```
I do not understand how is the code running well and using Nvidia Laptop GPU on my laptop where `GPU 0` is Intel UHD Graphics.
After disabling the AMD iGPU and enabling it again, the Windows Task manager on my Desktop shows as follows:
Desktop:
```
GPU 0
Nvidia GeForce RTX... Desktop GPU
Physical location: PCI bus 1, device 0, function 0
GPU 1
AMD Raedon...
Physical location: PCI bus 22, device 0, function 0
```
The code now works well on my desktop and uses the Nvidia Desktop GPU. But it did not work by default. I suppose that disabling and reenabling the iGPU changed the device ids.
Is there a way to query the device ids in my program and use the device that I want to use without any ambiguity?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.