microsoft / microsoft/onnxruntime
[Feature Request] make OrtSession#native run public or refactor to avoid array allocation on every inference
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
The public OrtSession#run function allocates multiple arrays for each call to inference. Likely in most programs the size of these arrays are never going to change. So either this function should be refactored to avoid allocation on every single inference or expose the native run function ( https://github.com/microsoft/onnxruntime/blob/main/java/src/main/java/ai/onnxruntime/OrtSession.java#L421 ) as public so that the caller map opt to avoid allocation
### Describe scenario use case
Avoid allocation of arrays for equivalent inferences.
Contributor guide
Research direction
Start in java/src/main/java/ai/onnxruntime/OrtSession.java, comparing the public run function with the native run function linked in the request. Determine whether the API should expose native run or avoid repeated array allocation for equivalent inferences; done means the selected approach removes the stated per-inference allocation concern.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100