microsoft / microsoft/onnxruntime

Run all Nodes on GPU/DML with DML-EP

Open
#21,013 3 comments 0 reactions 0 assignees View on GitHub
ep:DML feature request model:transformer platform:windows
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

I tried to run optimum models with DML EP (on my windows PC), for example take [optimum/vit-base-patch16-224 · Hugging Face](https://huggingface.co/optimum/vit-base-patch16-224)

model = ORTModelForImageClassification.from_pretrained(model_name, provider=“DmlExecutionProvider”)

onnx 1.16.1
onnxruntime 1.18.0
onnxruntime-directml 1.18.0
optimum 1.20.0

I see nodes are distributed between CPU EP & DML EP. Also, noticed different instances of same node are placed on both DML and CPU.

from verbose logs

2024-06-05 11:11:22.1833502 [V:onnxruntime:, session_state.cc:1152 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Node(s) placed on [DmlExecutionProvider]. Number of nodes: 335

2024-06-05 11:11:22.2061078 [V:onnxruntime:, session_state.cc:1154 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Concat (Concat_25)

2024-06-05 11:11:22.8286509 [V:onnxruntime:, session_state.cc:1152 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Node(s) placed on [CPUExecutionProvider]. Number of nodes: 9
2024-06-05 11:11:22.8322004 [V:onnxruntime:, session_state.cc:1154 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Concat (Concat_7)

For example take “Concat ” node/operator, I believe this node is supported on DML(Concat_25 - is placed on DML), then why Concat_7 instance of this node is placed on CPU

Why the few node instances are placed on CPU, even though DML have support for those nodes?

Here I mentioned Concat node as an example, in the full log I'm seeing the same behavior with other nodes Gather, Squeeze, Unsqueeze etc...

I expect, with provider=“DmlExecutionProvider” option, all nodes should be placed on DML only (exception - if there is no native support on DML for a particular node). But in the above case, all the nodes placed on CPU, support is present on DML

How can I force all nodes to be placed on DML? If the nodes got distributed b/w CPU and DML, I expect some overhead due to data transfer b/w CPU and DML

Thanks,

### Describe scenario use case

Trying the run the hugging face optimum model on GPU/DML with all noes on DML

Contributor guide

Open the contributing guide

Research direction

Reproduce the Hugging Face Optimum image-classification example with the listed ONNX Runtime and DML-EP versions, then inspect the verbose node-placement logs. Focus on why supported Concat, Gather, Squeeze, and Unsqueeze instances are split between DML and CPU. Done means explaining the placement behavior and providing or documenting a supported way to keep eligible nodes on DML.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.