Failed to utilise CUDA with TRT Engine when running on Jetson AGX Orin (ONNX->TRT, Transformer)

Description

I am trying to convert a DINO object detector Transformer trained with a custom dataset model to a TensorRT engine with any precision. I am using trtexec for that. The engine file is generated but it gives the following error:

[05/23/2023-10:55:09] [E] Error[1]: [executionContext.cpp::handleTrainStationRunnerPhase1::146] Error Code 1: Cuda Runtime (operation not permitted when stream is capturing)
[05/23/2023-10:55:09] [W] The CUDA graph capture on the stream has failed.
[05/23/2023-10:55:09] [W] The built TensorRT engine contains operations that are not permitted under CUDA graph capture mode.
[05/23/2023-10:55:09] [W] The specified --useCudaGraph flag has been ignored. The inference will be launched without using CUDA graph launch.

I can perform inference with a python script but I think it is not using the GPU for that and can only run at 3 FPS on Jetson Orin. I am expecting roughly at least 10 FPS on the Jetson Orin.
Any ideas what might be causing this issue and how I can solve that?
I used the following command for conversion:

trtexec --onnx=dino_simp.onnx --int8 --useCudaGraph --verbose --saveEngine=dino_last.trt --workspace=20000

Verbose output logs

Environment

TensorRT Version: 8.5.2

NVIDIA GPU: Jetson AGX ORIN

NVIDIA Driver Version: L4T 35.3.1

CUDA Version: 11.4.315

CUDNN Version: 8.6.0.166

Operating System: Ubuntu 20.04 LTS

Python Version (if applicable): 3.8

PyTorch Version (if applicable):

Container (if so, version): nvcr.io/nvidia/l4t-tensorrt:r8.5.2.2-devel

Relevant Files

Model link:
The model onnx file and the full verbose log output file can be downloaded at the following link: drive

Steps To Reproduce

Commands or scripts:
trtexec --onnx=dino_simp.onnx --int8 --useCudaGraph --verbose --saveEngine=dino_last.trt --workspace=20000

Have you tried the latest release?: The latest TensorRT release I can only try on my laptop but the corresponding Jetpack release is not yet available to be installed on the Jetson Orin. https://github.com/NVIDIA/TensorRT/issues/2949

Can this model run on other frameworks? I can do inference with ONNX runtime on my model. I also tried to convert the same model on my laptop and it works without any issues. I can do inference with about 14 FPS which is expected.

Thank you for looking into this.