[Feature Request] Thread-safe InferenceSession Initialization to Enable Parallel Session Scaling in Python NoGIL

Open

#27,089 opened on Jan 21, 2026 0 comments 0 reactions 0 assignees View on GitHub

feature request

Dominant language: C++
Stars: 21.9k
Forks: 4.2k
Avg merge: 4d 11h
Merged PRs (30d): 184

Description

### Describe the feature request

### Summary

I am requesting that the [InferenceSession](https://github.com/microsoft/onnxruntime/blob/c343143a49ca8fe39dd54c11a90febfa1cdd4d45/onnxruntime/python/onnxruntime_inference_collection.py#L430-L434) initialization process be made thread-safe, or alternatively, that its lack of thread-safety be explicitly documented/commented.

This is necessary because, for high-concurrency and low-batch workloads, maintaining thread-specific sessions (Scenario B) significantly outperforms sharing a single session (Scenario A) by avoiding runtime lock contention. Currently, initializing multiple sessions concurrently leads to race conditions, due to some internal functions such as [AddCustomOpDomains](https://github.com/microsoft/onnxruntime/blob/c343143a49ca8fe39dd54c11a90febfa1cdd4d45/onnxruntime/core/session/inference_session.h#L268)

### Benchmark test
In my benchmarks using ResNet-50 (just as an example) with Batch Size 1 on the TensorRT Execution Provider, I observed that:

- Scenario A (Shared Session): Multiple threads sharing one session suffer from internal runtime lock contention.

- Scenario B (Thread-specific Sessions): Each thread having its own session instance bypasses these locks, providing significantly higher total throughput.

Benchmark Environment:
- Model: ResNet-50 (ONNX)
- Execution Provider: TensorRT
- Python Version: 3.13 (Free-threaded/GIL-enabled)
- Hardware: Tesla T4

### Proposed Solutions

Option 1 (Preferred): Implement Internal Thread-Safety
- Ensure that the InferenceSession constructor are guarded by synchronization mechanisms such as global lock.
- This would allow users to scale their inference throughput by spawning sessions across multiple threads safely.

Option 2 : Documentation & Warning
- If architectural constraints prevent full thread-safety, please explicitly clarify that the initialization path is not thread-safe [here in the Python API](https://github.com/microsoft/onnxruntime/blob/c343143a49ca8fe39dd54c11a90febfa1cdd4d45/onnxruntime/python/onnxruntime_inference_collection.py#L430-L434).
- Add a warning or an internal check to prevent concurrent initialization attempts, informing users they must implement their own external synchronization (e.g., threading.Lock()).

Plus, I am happy to work on these.

benchmark test code

```python
import time
import threading
import os
import shutil
import argparse
import numpy as np
import onnxruntime as ort
import torch
import torchvision.models as models

MODEL_PATH = "resnet50.onnx"
TRT_CACHE_PATH = "./trt_cache_resnet"
ITERATIONS_PER_THREAD = 500

# 0. Cache init
def clear_cache():
if os.path.exists(TRT_CACHE_PATH):
shutil.rmtree(TRT_CACHE_PATH)
os.makedirs(TRT_CACHE_PATH, exist_ok=True)

# 1. ResNet-50 Model downlaod and ONNX Export
def prepare_model(batch_size):
if os.path.exists(MODEL_PATH):
print(f"Model {MODEL_PATH} already exists. Skipping download.")
return

model = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)
model.eval()
model.cuda()

input_shape = (batch_size, 3, 224, 224)
dummy_input = torch.randn(input_shape, device='cuda')

torch.onnx.export(
model,
dummy_input,
MODEL_PATH,
export_params=True,
opset_version=17,
do_constant_folding=True,
input_names=['input'],
output_names=['output'],
dynamic_axes={'input': {0: 'batch_size'}, 'output': {0: 'batch_size'}}
)
del model
torch.cuda.empty_cache()
print("Model export successful.")

# 2. TensorRT Provider Option setting
def get_trt_providers(batch_size):
input_tensor_name = 'input'
shape_str = f"{input_tensor_name}:{batch_size}x3x224x224"

trt_options = {
'trt_engine_cache_enable': True,
'trt_engine_cache_path': TRT_CACHE_PATH,
'trt_profile_min_shapes': shape_str,
'trt_profile_opt_shapes': shape_str,
'trt_profile_max_shapes': shape_str,
}

return [
('TensorrtExecutionProvider', trt_options),
]

# 3. Inference worker
def inference_worker(session, input_data, iterations, barrier=None):
input_name = session.get_inputs()[0].name
if barrier:
barrier.wait()

for _ in range(iterations):
session.run(None, {input_name: input_data})

# 4. Scenario A: Shared Session
def benchmark_shared_session(input_data, num_threads, batch_size):
print(f"\n--- [Scenario A] {num_threads} Threads Sharing 1 ResNet Session (batch={batch_size}) ---")

session = ort.InferenceSession(MODEL_PATH, providers=get_trt_providers(batch_size))

threads = []
barrier = threading.Barrier(num_threads + 1)

for _ in range(num_threads):
t = threading.Thread(target=inference_worker, args=(session, input_data, ITERATIONS_PER_THREAD, barrier))
threads.append(t)
t.start()

barrier.wait()
actual_start = time.time()
for t in threads:
t.join()
end_time = time.time()

total_time = end_time - actual_start
total_inferences = num_threads * ITERATIONS_PER_THREAD * batch_size
print(f"Total Time: {total_time:.4f} sec")
print(f"Throughput: {total_inferences / total_time:.2f} img/sec")

# 5. Scenario B: Separate Sessions
def benchmark_separate_sessions(input_data, num_threads, batch_size):
print(f"\n--- [Scenario B] {num_threads} Threads with Separate ResNet Sessions (batch={batch_size}) ---")

def thread_wrapper(input_data, iterations, barrier, batch_size):
local_session = ort.InferenceSession(MODEL_PATH, providers=get_trt_providers(batch_size))
inference_worker(local_session, input_data, iterations, barrier)

threads = []
barrier = threading.Barrier(num_threads + 1)

for _ in range(num_threads):
t = threading.Thread(target=thread_wrapper, args=(input_data, ITERATIONS_PER_THREAD, barrier, batch_size))
threads.append(t)
t.start()

print("Waiting for threads to initialize sessions...")
barrier.wait()
actual_start = time.time()
for t in threads:
t.join()
end_time = time.time()

total_time = end_time - actual_start
total_inferences = num_threads * ITERATIONS_PER_THREAD * batch_size
print(f"Total Time: {total_time:.4f} sec")
print(f"Throughput: {total_inferences / total_time:.2f} img/sec")

if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Benchmark ONNX inference with different thread counts")
parser.add_argument("--num-threads", type=int, default=4,
help="Number of threads to use for benchmarking (default: 4)")
parser.add_argument("--batch-size", type=int, default=1,
help="Batch size for inference (default: 1)")
args = parser.parse_args()

num_threads = args.num_threads
batch_size = args.batch_size
input_shape = (batch_size, 3, 224, 224)

print(f"=== Running benchmark with {num_threads} threads, batch_size={batch_size} ===")

clear_cache()
prepare_model(batch_size)

data = np.random.randn(*input_shape).astype(np.float32)
warmup_sess = ort.InferenceSession(MODEL_PATH, providers=get_trt_providers(batch_size))
warmup_sess.run(None, {'input': data})
del warmup_sess

benchmark_shared_session(data, num_threads, batch_size)
time.sleep(3)
benchmark_separate_sessions(data, num_threads, batch_size)

print("\nDone.")
```

benchmark results in a text file

```

########################################
BATCH SIZE: 1
########################################

========================================
Testing: threads=2, batch=1
========================================
=== Running benchmark with 2 threads, batch_size=1 ===
Model resnet50.onnx already exists. Skipping download.

--- [Scenario A] 2 Threads Sharing 1 ResNet Session (batch=1) ---
Total Time: 3.2840 sec
Throughput: 304.50 img/sec

--- [Scenario B] 2 Threads with Separate ResNet Sessions (batch=1) ---
Waiting for threads to initialize sessions...
Total Time: 3.2975 sec
Throughput: 303.26 img/sec

Done.