microsoft / microsoft/onnxruntime
[Performance] InferenceSession.run executes nodes which are unnecessary for requested outputs
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
I expected (and maybe that is the bug?) that the `output_names` argument of [`onnxruntime.InferenceSession.run`](https://onnxruntime.ai/docs/api/python/api_summary.html#onnxruntime.InferenceSession.run) is used to avoid unnecessary computations. However, it appears that the entire graph is executed even if only a subset of outputs is requested.
Avoiding potentially expensive and unnecessary computations would be very nice.
### To reproduce
The following code crashes (deliberately) in the `Cast` operation even though it is unnecessary for the computation of the requested "out_c" output. I attached the resulting ONNX graph as a file. I'm happy to type out a test with pure `onnx` functions rather than using [Spox](https://github.com/Quantco/spox) if required.
```python
from spox import argument, build, Tensor
import numpy as np
import onnxruntime as ort
import spox.opset.ai.onnx.v18 as op
def test_avoid_unecessary_compute():
a = argument(Tensor(float, ("N",)))
b = argument(Tensor(str, ("M",)))
c = op.add(a, a)
d = op.cast(b, to=np.int64)
model_proto = build(inputs={"a": a, "b": b}, outputs={"out_c": c, "out_d": d})
session = ort.InferenceSession(model_proto.SerializeToString())
session.run(["out_c"], {"a": [1.0], "b": ["foo"]}). # <= Crashes here
```
The raised error (unsurprisingly) reads:
```
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Cast node. Name:'Cast_0' Status Message: stoll: no conversion
```
### Urgency
We would like to use this feature in a future project where we would generate a large graph with different expensive-to-compute outputs. We would like to avoid building multiple models which only slightly differ in their set of outputs.
### Platform
Mac
### OS Version
MacOS 12.3.1
### ONNX Runtime Installation
Released Package
### ONNX Runtime Version or Commit ID
1.15.1
### ONNX Runtime API
Python
### Architecture
ARM64
### Execution Provider
Default CPU
### Execution Provider Library Version
_No response_
### Model File
[test.onnx.zip](https://github.com/microsoft/onnxruntime/files/11871714/test.onnx.zip)
### Is this a quantized model?
No
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Python InferenceSession.run entry point and the linked API documentation, then reproduce the attached test.onnx.zip case using the provided output_names and inputs. Trace how requested outputs are selected and verify that nodes unrelated to out_c are not executed; done means the example returns out_c without raising the Cast conversion error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100