microsoft / microsoft/onnxruntime-genai

Using Onnxruntime Custom Execution Provider for ORT-GenAI

Open
#2,006 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

In Onnxruntime, we are able to create custom Execution Providers via either shared provider or Plugin EP API.
Is there a way to integrate such custom EPs to onnxruntime-genai?
I only see CPU, QNN and CUDA and now WebGPU EP for onnxruntime-genai.

How can we integrate custom execution providers built via onnxruntime to onnxruntime-genai, since ORT-GenAI does not seem to recognize standalone ORT EPs. Am i missing something?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing how ORT-GenAI currently exposes CPU, QNN, CUDA, and WebGPU execution providers, then compare that with the shared provider and Plugin EP APIs mentioned in the issue. Determine whether standalone ONNX Runtime custom providers are intended to be supported and document the integration path or the missing design needed to make them available.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai-infra-agents
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.