microsoft / microsoft/onnxruntime

[Feature Request] Make DirectML execution provider thread safe (allow Run() concurrency)

Open
#22,147 3 comments 0 reactions 0 assignees View on GitHub
ep:DML feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

The DirectML execution provider is not currenly thread-safe, which means that if you have any concurrency with a session on the Run() call (multiple threads calling Run on same session) it will crash.

This goes directly agains the high-level design of ORT and means that users must create N session to achieve N concurrency, which dramatically increases the memory requirements.
Alghough the problem is documented, it is easy to miss as there is a lot of material promoting the use of a single session instance for concurrency.

https://onnxruntime.ai/docs/execution-providers/DirectML-ExecutionProvider.html
"Additionally, as the DirectML execution provider does not support parallel execution, it does not support multi-threaded calls to Run on the same inference session. That is, if an inference session using the DirectML execution provider, only one thread may call Run at a time. Multiple threads are permitted to call Run simultaneously if they operate on different inference session objects."

What are the reasons the EP is not already thread safe?
For me it crashes in the bucketized allocator.
Are there any workarounds?

### Describe scenario use case

It would be quite beneficial for any users that use DML EP and that also need concurrency to be able to use a single inference session.
- Reduced memory cost.
- Reduced initialization cost (no need to create N inference sessions).

Contributor guide

Open the contributing guide

Research direction

Start with the DirectML execution provider's Run() path and the bucketized allocator mentioned in the report. Review the documented single-thread limitation and the existing issue discussion before investigating the concurrency design. Done means concurrent Run() calls on one inference session no longer crash and the behavior is covered by an appropriate test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.