microsoft / microsoft/onnxruntime

Null Tensor as output when using TensorRT without session io bindings

Open
#18,509 15 comments 0 reactions 1 assignee Claimed by @chilo-ms View on GitHub
ep:CUDA ep:TensorRT platform:windows
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

When I use the TensorRT execution provider and I use session.run like this:
'''
std::vector lOutputTensors = mOrtSession->Run(Ort::RunOptions{ nullptr }, mInputNodeNamePointers.data(), lInputTensors.data(),
mInputNodeNamePointers.size(), mOutputNodeNamePointers.data(), mOutputNodeNamePointers.size());
'''
Then I get errors when calling lOutputTensors[0].GetTensorMutableData() saying that the output is null. This does not happen when I use the cuda execution provider. Also when I use session io bindings with TensorRT then I have no issues. But there are some reasons that lead to me not wanting to use session io bindings, so I would like it to work without them. I did found someone online who seemingly got it to work without session io bindings, but I saw that he was using an experimental version of onnxruntime:
https://github.com/microsoft/onnxruntime/issues/14614

So I guess a follow up question is, if it is necessary how do I build the experimental version of onnxruntime?

### To reproduce

Using TensorRT as your execution provider and then calling session.run() like this:
'''
std::vector lOutputTensors = mOrtSession->Run(Ort::RunOptions{ nullptr }, mInputNodeNamePointers.data(), lInputTensors.data(),
mInputNodeNamePointers.size(), mOutputNodeNamePointers.data(), mOutputNodeNamePointers.size());
'''
Followed by lOutputTensors[0].GetTensorMutableData()

### Urgency

_No response_

### Platform

Windows

### OS Version

10.0.19045

### ONNX Runtime Installation

Built from Source

### ONNX Runtime Version or Commit ID

1.16.1

### ONNX Runtime API

C++

### Architecture

X64

### Execution Provider

TensorRT

### Execution Provider Library Version

8.6.1.6

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.