microsoft / microsoft/onnxruntime
[Feature Request] Enable and Disable profiling for warm up
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
I would like to do some profiling with `onnxruntime` using the utilities described here [here](https://onnxruntime.ai/docs/api/python/auto_examples/plot_profiling.html).
But following this tutorial, I can only set `session_options.enable_profiling=True` before the inference session starts and end the profiling with `session.end_profiling()` once and for all.
I want to be able to:
- warm up the session/model before the profiling starts.
- compute other kind other statistics like `stddev` of a kernel/node's runtime.
This can be achieved by having a different profiling file each time `session.end_profiling()` is called like in this example:
```python
from optimum.onnxruntime import ORTModelForSequenceClassification
from tempfile import TemporaryDirectory
import onnxruntime as ort
import pandas as pd
import numpy
inputs = {
'input_ids': numpy.array([[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]]),
'attention_mask': numpy.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]),
'token_type_ids': numpy.array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]),
}
with TemporaryDirectory() as tmpdirname:
session_options = ort.SessionOptions()
session_options.enable_profiling = True
session_options.profile_file_prefix = tmpdirname + '/profiling'
model = ORTModelForSequenceClassification.from_pretrained(
"bert-base-uncased",
export=True,
session_options=session_options,
)
for _ in range(5):
model(**inputs)
warmup_file = model.model.end_profiling()
# after this maybe a new profiling file can be created
for _ in range(10):
model(**inputs)
profiling_file = model.model.end_profiling()
profiling_df = pd.read_json(profiling_file, orient='records')
# processing profiling_df
```
### Describe scenario use case
stable and in-depth profiling with no outliers.
Contributor guide
Research direction
Start with the Python profiling example linked in the issue and the SessionOptions.enable_profiling and session.end_profiling APIs. Trace their implementation and existing profiling tests, if any; done should include a defined way to warm up before profiling and produce separate profiling data for later runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100