microsoft / microsoft/onnxruntime
[Feature Request] Make DirectML execution provider thread safe (allow Run() concurrency)
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
The DirectML execution provider is not currenly thread-safe, which means that if you have any concurrency with a session on the Run() call (multiple threads calling Run on same session) it will crash.
This goes directly agains the high-level design of ORT and means that users must create N session to achieve N concurrency, which dramatically increases the memory requirements.
Alghough the problem is documented, it is easy to miss as there is a lot of material promoting the use of a single session instance for concurrency.
https://onnxruntime.ai/docs/execution-providers/DirectML-ExecutionProvider.html
"Additionally, as the DirectML execution provider does not support parallel execution, it does not support multi-threaded calls to Run on the same inference session. That is, if an inference session using the DirectML execution provider, only one thread may call Run at a time. Multiple threads are permitted to call Run simultaneously if they operate on different inference session objects."
What are the reasons the EP is not already thread safe?
For me it crashes in the bucketized allocator.
Are there any workarounds?
### Describe scenario use case
It would be quite beneficial for any users that use DML EP and that also need concurrency to be able to use a single inference session.
- Reduced memory cost.
- Reduced initialization cost (no need to create N inference sessions).
Contributor guide
Research direction
Start with the DirectML execution provider's Run() path and the bucketized allocator mentioned in the report. Review the documented single-thread limitation and the existing issue discussion before investigating the concurrency design. Done means concurrent Run() calls on one inference session no longer crash and the behavior is covered by an appropriate test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100