microsoft / microsoft/onnxruntime

[Documentation/Performance] Parallelize model execution by chunking batch dimension

Open
#17,349 1 comment 0 reactions 0 assignees View on GitHub
documentation
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the documentation issue

Some models have a "batch dimension" in their inputs suggesting that entries along that dimension are independent of each other. Models of this kind are good candidates for embarrassingly parallel execution: Simply chunk the inputs along that dimension, execute each chunk in its own thread, and lastly concatenate the outputs. A simple parallelization model of this kind can better utilize the available hardware in some use-cases.

While onnxruntime has two prominent options for parallelization (`intra_op_num_threads` and `inter_op_num_threads` of the `SessionOptions` object) I did not find any documentation of this kind of parallelization. It appears to me that an embarrassingly parallel approach would have significant advantages over the aforementioned options as I understand them. Did I miss the possibility to somehow communicate to onnxruntime that a certain dimension is a batch dimension to be used for parallelization, or does that feature simply not exist?

### Page / URL

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing ONNX Runtime's SessionOptions, especially intra_op_num_threads and inter_op_num_threads, and how model execution handles independent batch entries. Determine whether batch-dimension chunking is supported or configurable. Done means documenting the existing behavior and configuration, or clearly recording that this parallelization model is not available.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, machine-learning
Domain
machine-learning, performance
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.