microsoft / microsoft/onnxruntime

[Feature Request] Enable and Disable profiling for warm up

Open
#15,980 1 comment 0 reactions 0 assignees View on GitHub
core runtime feature request model:transformer
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

I would like to do some profiling with `onnxruntime` using the utilities described here [here](https://onnxruntime.ai/docs/api/python/auto_examples/plot_profiling.html).
But following this tutorial, I can only set `session_options.enable_profiling=True` before the inference session starts and end the profiling with `session.end_profiling()` once and for all.

I want to be able to:
- warm up the session/model before the profiling starts.
- compute other kind other statistics like `stddev` of a kernel/node's runtime.

This can be achieved by having a different profiling file each time `session.end_profiling()` is called like in this example:

```python
from optimum.onnxruntime import ORTModelForSequenceClassification
from tempfile import TemporaryDirectory
import onnxruntime as ort
import pandas as pd
import numpy

inputs = {
'input_ids': numpy.array([[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]]),
'attention_mask': numpy.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]),
'token_type_ids': numpy.array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]),
}

with TemporaryDirectory() as tmpdirname:
session_options = ort.SessionOptions()
session_options.enable_profiling = True
session_options.profile_file_prefix = tmpdirname + '/profiling'
model = ORTModelForSequenceClassification.from_pretrained(
"bert-base-uncased",
export=True,
session_options=session_options,
)

for _ in range(5):
model(**inputs)

warmup_file = model.model.end_profiling()
# after this maybe a new profiling file can be created

for _ in range(10):
model(**inputs)

profiling_file = model.model.end_profiling()
profiling_df = pd.read_json(profiling_file, orient='records')
# processing profiling_df
```

### Describe scenario use case

stable and in-depth profiling with no outliers.

Contributor guide

Open the contributing guide

Research direction

Start with the Python profiling example linked in the issue and the SessionOptions.enable_profiling and session.end_profiling APIs. Trace their implementation and existing profiling tests, if any; done should include a defined way to warm up before profiling and produce separate profiling data for later runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.