microsoft / microsoft/onnxruntime-genai

C# Run Program on NPU (OnnxRuntime + DirectML + NPU)?

Open
#1,189 2 comments 0 reactions 0 assignees View on GitHub
ep:DML
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

Hi, big guys!
Recently I tried to run phi3 on NPU inference using DirectML as backend EP under Onnxruntime framework, but I failed!
I loaded the model and tokenizer through the Model class and Tokenizer class, but from the C# API given in the official Onnxruntime docs(https://onnxruntime.ai/docs/genai/api/csharp.html), I can't find any API about being able to set up the NPU, can someone expand on this in detail to explain how to load the model onto the NPU?(It's best if you can explain it through a demo)
Or instead do you know if there are demos on loading Phi3 or other LLM models to NPU?
Looking forward to your reply!
Below is the code I used to load the model and tokenizer using the OnnxruntimeGenai-directml library and enter the question to be answered.

![Image](https://github.com/user-attachments/assets/8ba633de-0639-4303-ae75-018f1a0188d6)

![Image](https://github.com/user-attachments/assets/f0beaee4-737a-4550-802b-ea5141b5b6df)

![Image](https://github.com/user-attachments/assets/e7707733-bc65-4b37-8c1a-d67feb0ab012)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked ONNX Runtime GenAI C# API documentation and the reported OnnxruntimeGenai-directml usage. Determine whether the existing API supports selecting an NPU through DirectML, then document the supported setup or provide a Phi-3/NPU demonstration; completion should make the required configuration and expected result clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
ai, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.