microsoft / microsoft/onnxruntime
[Feature Request] Multi-Shape Profiling in Perftest Tool
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
## Summary
Extend the ORT perftest tool (`onnxruntime_perf_test`) to support profiling multiple input shapes within a **single session**. This enables more realistic benchmarking of models with dynamic dimensions: load model once and run inference with varying shapes, which is how applications use dynamic models.
## Motivation
Currently, benchmarking multiple shapes requires running the perftest tool in a loop:
```bash
for batch in 1 2 4 8 16 32; do
echo "=== Batch size: $batch ==="
./onnxruntime_perf_test -m times -r 100 -I -f "batch:$batch" model.onnx
done
```
Each invocation creates a **separate session**, which:
- May not reflect real-world usage (applications typically load the model once, run many shapes).
- Prevents capturing cross-shape aggregate statistics.
OpenVINO's benchmark app already provides this capability via `-data_shape`, producing per-shape and overall latency reports.
## Expected Behavior
### Example CLI usage
```bash
./onnxruntime_perf_test -m times -r 300 -I \
-data_shape "input:[1,16,1440,2560][1,16,1080,1920][1,16,720,1280]" \
model.onnx
```
### Example output
Here is example output from the OpenVINO benchmarking app. Something similar can be done for the ORT perftest tool.
```
Overall Latency:
Median: 9.85 ms
Average: 10.56 ms
Min: 4.21 ms
Max: 18.99 ms
Latency per shape group:
1. input : [1,16,1440,2560]
Median: 17.04 ms
Average: 17.08 ms
Min: 16.32 ms
Max: 18.99 ms
2. input : [1,16,1080,1920]
Median: 9.85 ms
Average: 9.96 ms
Min: 9.46 ms
Max: 11.45 ms
3. input : [1,16,720,1280]
Median: 4.52 ms
Average: 4.63 ms
Min: 4.21 ms
Max: 6.85 ms
```
## Requirements
- Input data should be **auto-generated** by the perftest tool (when `-I` is provided) for each shape group.
- If `-I` is not provided, then perhaps it should be an error? Or `-I` is implied?
- Latency should be reported **separately** for each shape group.
- **Overall** latency across all shapes should also be reported.
- The model is loaded **once**; all shapes run within the same `Ort::Session`.
## Implementation Details
### What It Would Take
#### 1. New CLI argument (`command_args_parser.cc`)
- Add a `-data_shape` flag (or similar) that accepts one or more shape specifications.
- Syntax: `"input_name:[d0,d1,...][d0,d1,...]"`; multiple bracket groups per input, multiple inputs separated by spaces.
- Store parsed shapes in `RunConfig` as `std::map>>`.
#### 2. Extend input data generation (`ort_test_session.cc` - `PopulateGeneratedInputTestData`)
- Currently generates a single test data set (id=0) using the model's metadata shape (free dims become 1).
- Needs to generate **N** test data sets (one per shape group), using the user-provided dimensions for each group.
- Each shape group populates a separate `test_inputs_[i]` slot via `PreLoadTestData(i, ...)`.
#### 3. Modify iteration logic (`ort_test_session.cc` - `Run()`)
- Currently selects a random `test_inputs_` entry per iteration.
- With multi-shape mode, cycle through shape groups deterministically (e.g., round-robin) to ensure equal coverage.
- Return or tag the shape group index used for each iteration so timing can be attributed.
#### 4. Extend result tracking (`performance_runner.h/cc`)
- Add per-shape-group timing vectors to `PerformanceResult` (e.g., `std::vector> per_shape_time_costs`).
- In `RunOneIteration`, record which shape group was used and store the timing in the correct bucket.
- Compute and print per-group statistics (median, average, min, max) in addition to the existing overall stats.
## Key Design Decisions
### 1. Argument format
**Option A**: OpenVINO-style `-data_shape "input_name:[shape1][shape2]..."` — familiar to users of other tools, explicit per-input.
**Option B**: Extend existing `-f` (free dimension override) to accept multiple values — e.g., `-f "batch:1,2,4,8"`. Less verbose but only works for named free dimensions.
**Recommendation**: Option A is more general and handles arbitrary shape changes (not just named free dims). Option B could be a simpler first step for the common "vary one dimension" case.
### 2. Shape cycling strategy
**Option A**: Round-robin — each iteration advances to the next shape group. Ensures equal coverage and predictable order.
**Option B**: Random selection — current behavior for multiple test data sets. Stresses cache/memory differently.
**Option C**: Sequential blocks — run N iterations per shape, then move to the next. Cleanest for reporting but less realistic.
**Recommendation**: Round-robin (Option A) as default — equal coverage, deterministic, easy to reason about. Could offer random as an optional mode.
### 3. Multi-input models
Some models have multiple dynamic inputs. The format should support specifying shapes for each input independently:
```
-data_shape "input1:[1,3,224,224][1,3,448,448]" -data_shape "input2:[1,10][1,20]"
```
All inputs must have the same number of shape groups (enforced at parse time).
### 4. Interaction with existing `-f` flag
When both `-f` (free dim overrides) and `-data_shape` are specified:
- `-data_shape` takes precedence for the specified inputs.
- `-f` still applies to any inputs/dimensions not covered by `-data_shape`.
- If conflicting, emit a clear error message.
### 5. Warmup handling
Each shape group should receive at least one warmup iteration before measurement begins, since the first run of a new shape may trigger EP-specific compilation or memory allocation.
### Describe scenario use case
This enables more realistic benchmarking of models with dynamic dimensions: load model once and run inference with varying shapes, which is how applications use dynamic models.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with command_args_parser.cc and RunConfig to trace how perftest arguments are parsed, then inspect PopulateGeneratedInputTestData and Run in ort_test_session.cc. Review performance_runner.h/cc for timing aggregation. Done means one Ort::Session can run the requested shape groups, with separate per-shape and overall latency statistics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- cli, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100