microsoft / microsoft/onnxruntime
Run all Nodes on GPU/DML with DML-EP
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
I tried to run optimum models with DML EP (on my windows PC), for example take [optimum/vit-base-patch16-224 · Hugging Face](https://huggingface.co/optimum/vit-base-patch16-224)
model = ORTModelForImageClassification.from_pretrained(model_name, provider=“DmlExecutionProvider”)
onnx 1.16.1
onnxruntime 1.18.0
onnxruntime-directml 1.18.0
optimum 1.20.0
I see nodes are distributed between CPU EP & DML EP. Also, noticed different instances of same node are placed on both DML and CPU.
from verbose logs
2024-06-05 11:11:22.1833502 [V:onnxruntime:, session_state.cc:1152 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Node(s) placed on [DmlExecutionProvider]. Number of nodes: 335
2024-06-05 11:11:22.2061078 [V:onnxruntime:, session_state.cc:1154 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Concat (Concat_25)
2024-06-05 11:11:22.8286509 [V:onnxruntime:, session_state.cc:1152 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Node(s) placed on [CPUExecutionProvider]. Number of nodes: 9
2024-06-05 11:11:22.8322004 [V:onnxruntime:, session_state.cc:1154 onnxruntime::VerifyEachNodeIsAssignedToAnEp] Concat (Concat_7)
For example take “Concat ” node/operator, I believe this node is supported on DML(Concat_25 - is placed on DML), then why Concat_7 instance of this node is placed on CPU
Why the few node instances are placed on CPU, even though DML have support for those nodes?
Here I mentioned Concat node as an example, in the full log I'm seeing the same behavior with other nodes Gather, Squeeze, Unsqueeze etc...
I expect, with provider=“DmlExecutionProvider” option, all nodes should be placed on DML only (exception - if there is no native support on DML for a particular node). But in the above case, all the nodes placed on CPU, support is present on DML
How can I force all nodes to be placed on DML? If the nodes got distributed b/w CPU and DML, I expect some overhead due to data transfer b/w CPU and DML
Thanks,
### Describe scenario use case
Trying the run the hugging face optimum model on GPU/DML with all noes on DML
Contributor guide
Research direction
Reproduce the Hugging Face Optimum image-classification example with the listed ONNX Runtime and DML-EP versions, then inspect the verbose node-placement logs. Focus on why supported Concat, Gather, Squeeze, and Unsqueeze instances are split between DML and CPU. Done means explaining the placement behavior and providing or documenting a supported way to keep eligible nodes on DML.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100