microsoft / microsoft/onnxruntime

[Feature Request] Global Threadpool in Python API

Open
#23,523 10 comments 1 reaction 0 assignees View on GitHub
feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

Expose the ability to utilize a global threadpool for inference sessions in the python API

### Describe scenario use case

My current use case requires the instantiation of many (thousands) of small onnx models in memory at once. Doing so causes too many threads to be spawned halting the program. The functionality for a global threadpool exists in the cpp source but is not exposed to the python bindings.

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing global threadpool implementation in the C++ source and the Python binding entry points for inference sessions. Trace how session threadpools are currently configured, then define and test the Python-facing behavior needed to reuse one pool across many sessions.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
api, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.