microsoft / microsoft/onnxruntime-genai

Receiving ROCm execution provider not supported in this build

Open
#1,130 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

**Describe the bug**
Hi team,
I am currently building OnnxRuntime-GenAI from source for AMD GPUs. With some tweaks to the main branch, I was able to successfully build the `wheel` and install the python package. I was able to use the Model builder to convert LLMs to ROCm supported models. But when I try to run the inference, I am seeing the following error -

```shell
onnxruntime_genai.onnxruntime_genai.OrtException: ROCm execution provider is not enabled in this build.
```
I am sure there is no issue with onnxruntime dependency because I received the following supported EPs from it -

```shell
/workspace# python -c "import onnxruntime as ort; print(ort.get_available_providers())"
['MIGraphXExecutionProvider', 'ROCMExecutionProvider', 'CPUExecutionProvider']
```

Please suggest how I can solve this problem.

**To Reproduce**
Steps to reproduce the behavior:
1. Build and Install OnnxRuntime-Genai from source on a machine with AMD GPU.

**Expected behavior**
I expect no errors and LLMs should output same as they do in Nvidia GPUs.

**Desktop (please complete the following information):**
- OS: RunPod - Ubuntu 22.04 - ROCm 6.1

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the source build on Ubuntu 22.04 with ROCm 6.1 using the steps in the issue, then inspect the build configuration and inference setup for how execution providers are enabled. Confirm that the installed package can use the ROCm execution provider and that the reported inference error no longer occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux, python, ubuntu
Domain
build-system, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.