microsoft / microsoft/onnxruntime

[Feature Request] make OrtSession#native run public or refactor to avoid array allocation on every inference

Open
#13,574 7 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

The public OrtSession#run function allocates multiple arrays for each call to inference. Likely in most programs the size of these arrays are never going to change. So either this function should be refactored to avoid allocation on every single inference or expose the native run function ( https://github.com/microsoft/onnxruntime/blob/main/java/src/main/java/ai/onnxruntime/OrtSession.java#L421 ) as public so that the caller map opt to avoid allocation

### Describe scenario use case

Avoid allocation of arrays for equivalent inferences.

Contributor guide

Open the contributing guide

Research direction

Start in java/src/main/java/ai/onnxruntime/OrtSession.java, comparing the public run function with the native run function linked in the request. Determine whether the API should expose native run or avoid repeated array allocation for equivalent inferences; done means the selected approach removes the stated per-inference allocation concern.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.