microsoft / microsoft/onnxruntime
[Feature Request] Global Threadpool in Python API
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
Expose the ability to utilize a global threadpool for inference sessions in the python API
### Describe scenario use case
My current use case requires the instantiation of many (thousands) of small onnx models in memory at once. Doing so causes too many threads to be spawned halting the program. The functionality for a global threadpool exists in the cpp source but is not exposed to the python bindings.
Contributor guide
Research direction
Start by locating the existing global threadpool implementation in the C++ source and the Python binding entry points for inference sessions. Trace how session threadpools are currently configured, then define and test the Python-facing behavior needed to reuse one pool across many sessions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- api, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100