microsoft / microsoft/onnxruntime-genai

Cuda / DirectML question

Open
#1,037 6 comments 0 reactions 0 assignees View on GitHub
ep:CUDA ep:DML
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

Hi there, you made fantastic framework for llms. But what I find very confusing is how to run this on cuda and direct ml. I simply don't know how to do it in C#..

I there any example? Second question is, do I have to provide different model per cuda, cpu and directml or can it run seamlessly? Or is there a way to convert model to support all or combination of providers? as far as I know onnx it self provides seamles support that's why it's a bit confusing.

My use case is to deploy to user's device a model and based on his capabilities to choose the provider which can provide best performance. But not in the opposite direction, because I expect my user to know nothing about the ML it self.

Thank you ✌️

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names no repository files, tests, or entry points. First determine whether C# examples and provider-selection guidance belong in the repository, then document the supported CUDA, CPU, and DirectML workflow and verify the examples against the relevant runtime setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, csharp
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.