microsoft / microsoft/onnxruntime

[Feature Request] Multi-Shape Profiling in Perftest Tool

Open
#28,628 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

ep:OpenVINO feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

## Summary

Extend the ORT perftest tool (`onnxruntime_perf_test`) to support profiling multiple input shapes within a **single session**. This enables more realistic benchmarking of models with dynamic dimensions: load model once and run inference with varying shapes, which is how applications use dynamic models.

## Motivation

Currently, benchmarking multiple shapes requires running the perftest tool in a loop:

```bash
for batch in 1 2 4 8 16 32; do
echo "=== Batch size: $batch ==="
./onnxruntime_perf_test -m times -r 100 -I -f "batch:$batch" model.onnx
done
```

Each invocation creates a **separate session**, which:
- May not reflect real-world usage (applications typically load the model once, run many shapes).
- Prevents capturing cross-shape aggregate statistics.

OpenVINO's benchmark app already provides this capability via `-data_shape`, producing per-shape and overall latency reports.

## Expected Behavior

### Example CLI usage

```bash
./onnxruntime_perf_test -m times -r 300 -I \
-data_shape "input:[1,16,1440,2560][1,16,1080,1920][1,16,720,1280]" \
model.onnx
```

### Example output
Here is example output from the OpenVINO benchmarking app. Something similar can be done for the ORT perftest tool.

```
Overall Latency:
Median: 9.85 ms
Average: 10.56 ms
Min: 4.21 ms
Max: 18.99 ms

Latency per shape group:
1. input : [1,16,1440,2560]
Median: 17.04 ms
Average: 17.08 ms
Min: 16.32 ms
Max: 18.99 ms
2. input : [1,16,1080,1920]
Median: 9.85 ms
Average: 9.96 ms
Min: 9.46 ms
Max: 11.45 ms
3. input : [1,16,720,1280]
Median: 4.52 ms
Average: 4.63 ms
Min: 4.21 ms
Max: 6.85 ms
```

## Requirements

- Input data should be **auto-generated** by the perftest tool (when `-I` is provided) for each shape group.
- If `-I` is not provided, then perhaps it should be an error? Or `-I` is implied?
- Latency should be reported **separately** for each shape group.
- **Overall** latency across all shapes should also be reported.
- The model is loaded **once**; all shapes run within the same `Ort::Session`.

## Implementation Details

### What It Would Take

#### 1. New CLI argument (`command_args_parser.cc`)
- Add a `-data_shape` flag (or similar) that accepts one or more shape specifications.
- Syntax: `"input_name:[d0,d1,...][d0,d1,...]"`; multiple bracket groups per input, multiple inputs separated by spaces.
- Store parsed shapes in `RunConfig` as `std::map>>`.

#### 2. Extend input data generation (`ort_test_session.cc` - `PopulateGeneratedInputTestData`)
- Currently generates a single test data set (id=0) using the model's metadata shape (free dims become 1).
- Needs to generate **N** test data sets (one per shape group), using the user-provided dimensions for each group.
- Each shape group populates a separate `test_inputs_[i]` slot via `PreLoadTestData(i, ...)`.

#### 3. Modify iteration logic (`ort_test_session.cc` - `Run()`)
- Currently selects a random `test_inputs_` entry per iteration.
- With multi-shape mode, cycle through shape groups deterministically (e.g., round-robin) to ensure equal coverage.
- Return or tag the shape group index used for each iteration so timing can be attributed.

#### 4. Extend result tracking (`performance_runner.h/cc`)
- Add per-shape-group timing vectors to `PerformanceResult` (e.g., `std::vector> per_shape_time_costs`).
- In `RunOneIteration`, record which shape group was used and store the timing in the correct bucket.
- Compute and print per-group statistics (median, average, min, max) in addition to the existing overall stats.

## Key Design Decisions

### 1. Argument format
**Option A**: OpenVINO-style `-data_shape "input_name:[shape1][shape2]..."` — familiar to users of other tools, explicit per-input.

**Option B**: Extend existing `-f` (free dimension override) to accept multiple values — e.g., `-f "batch:1,2,4,8"`. Less verbose but only works for named free dimensions.

**Recommendation**: Option A is more general and handles arbitrary shape changes (not just named free dims). Option B could be a simpler first step for the common "vary one dimension" case.

### 2. Shape cycling strategy
**Option A**: Round-robin — each iteration advances to the next shape group. Ensures equal coverage and predictable order.

**Option B**: Random selection — current behavior for multiple test data sets. Stresses cache/memory differently.

**Option C**: Sequential blocks — run N iterations per shape, then move to the next. Cleanest for reporting but less realistic.

**Recommendation**: Round-robin (Option A) as default — equal coverage, deterministic, easy to reason about. Could offer random as an optional mode.

### 3. Multi-input models
Some models have multiple dynamic inputs. The format should support specifying shapes for each input independently:
```
-data_shape "input1:[1,3,224,224][1,3,448,448]" -data_shape "input2:[1,10][1,20]"
```
All inputs must have the same number of shape groups (enforced at parse time).

### 4. Interaction with existing `-f` flag
When both `-f` (free dim overrides) and `-data_shape` are specified:
- `-data_shape` takes precedence for the specified inputs.
- `-f` still applies to any inputs/dimensions not covered by `-data_shape`.
- If conflicting, emit a clear error message.

### 5. Warmup handling
Each shape group should receive at least one warmup iteration before measurement begins, since the first run of a new shape may trigger EP-specific compilation or memory allocation.

### Describe scenario use case

This enables more realistic benchmarking of models with dynamic dimensions: load model once and run inference with varying shapes, which is how applications use dynamic models.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with command_args_parser.cc and RunConfig to trace how perftest arguments are parsed, then inspect PopulateGeneratedInputTestData and Run in ort_test_session.cc. Review performance_runner.h/cc for timing aggregation. Done means one Ort::Session can run the requested shape groups, with separate per-shape and overall latency statistics.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli, machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.