microsoft / microsoft/onnxruntime
[Feature Request] ORT-Profiler: Include timestamps for tensor allocations and deallocations.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
I'm currently working with ONNXRuntime for performance-critical applications in Java, and I've found it challenging to optimize memory usage without detailed insights into tensor allocation lifetimes. In TensorFlow, I am accustomed to using the profiler to obtain metadata about tensor allocations, such as allocation/deallocation timestamps and bytes allocated.
More information about tensorflow profiler can be found here: [RunMetadata](https://github.com/tensorflow/tensorflow/blob/8a20d54a3c1bfa38c03ea99a2ad3c1b0a45dfa95/tensorflow/core/protobuf/config.proto#L771C9-L801) and [StepStats](https://github.com/tensorflow/tensorflow/blob/r2.9/tensorflow/core/framework/step_stats.proto)
Given the above information, allocation time can be inferred for each tensor.
Is there similar profiling capabilities that allow us to track the lifetime of tensor allocations in ONNXRuntime?
Using [SessionOptions#enableProfiling](https://onnxruntime.ai/docs/api/java/ai/onnxruntime/OrtSession.SessionOptions.html#enableProfiling(java.lang.String)) gives no such information.
### Describe scenario use case
This information is crucial for identifying bottlenecks and optimizing the memory footprint of models during inference or training.
Such a feature should provide:
- Timestamps for tensor allocations and deallocations.
- The size (in bytes) of each tensor allocation.
- Peak memory usage statistics.
- Live bytes currently allocated.
Contributor guide
Research direction
Start with the Java entry point SessionOptions#enableProfiling and inspect the profiling data it currently exposes. Compare that output with the linked TensorFlow RunMetadata and StepStats definitions, then determine the scope needed for allocation timestamps, sizes, peak usage, and live bytes. Done means the feature has a clear design and implementation path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100