microsoft / microsoft/onnxruntime

[Feature Request] ORT-Profiler: Include timestamps for tensor allocations and deallocations.

Open
#21,337 2 comments 0 reactions 0 assignees View on GitHub
api:Java feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

I'm currently working with ONNXRuntime for performance-critical applications in Java, and I've found it challenging to optimize memory usage without detailed insights into tensor allocation lifetimes. In TensorFlow, I am accustomed to using the profiler to obtain metadata about tensor allocations, such as allocation/deallocation timestamps and bytes allocated.

More information about tensorflow profiler can be found here: [RunMetadata](https://github.com/tensorflow/tensorflow/blob/8a20d54a3c1bfa38c03ea99a2ad3c1b0a45dfa95/tensorflow/core/protobuf/config.proto#L771C9-L801) and [StepStats](https://github.com/tensorflow/tensorflow/blob/r2.9/tensorflow/core/framework/step_stats.proto)

Given the above information, allocation time can be inferred for each tensor.

Is there similar profiling capabilities that allow us to track the lifetime of tensor allocations in ONNXRuntime?

Using [SessionOptions#enableProfiling](https://onnxruntime.ai/docs/api/java/ai/onnxruntime/OrtSession.SessionOptions.html#enableProfiling(java.lang.String)) gives no such information.

### Describe scenario use case

This information is crucial for identifying bottlenecks and optimizing the memory footprint of models during inference or training.

Such a feature should provide:
- Timestamps for tensor allocations and deallocations.
- The size (in bytes) of each tensor allocation.
- Peak memory usage statistics.
- Live bytes currently allocated.

Contributor guide

Open the contributing guide

Research direction

Start with the Java entry point SessionOptions#enableProfiling and inspect the profiling data it currently exposes. Compare that output with the linked TensorFlow RunMetadata and StepStats definitions, then determine the scope needed for allocation timestamps, sizes, peak usage, and live bytes. Done means the feature has a clear design and implementation path.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.