Internal Error (Could not find any implementation for node model/backbones.0/backbones.0.0/body/conv1/Conv + onnx::Mul_3979 + model/backbones.0/backbones.0.0/body/bn1/Mul + onnx::Add_3981 + model/backbones.0/backbones.0.0/body/bn1/Add + model/backbones.0/backbones.0.0/body/relu/Relu.

Description

Attempting to convert an ONNX model to a TensorRT engine using the trtexec tool fails, with errors indicating that no kernel image is available for the device and that no implementation can be found for the model's convolutional layer and subsequent batch normalization/activation layers. Specific errors include:

• no kernel image is available for execution on the device

• Could not find any implementation for node model/backbones.0/backbones.0.0/body/conv1/Conv + ... + Relu.

Environment

TensorRT Version: 10.3.0.30 (arm64 architecture, compatible with CUDA 12.6)
NVIDIA GPU: NVIDIA Jetson AGX Orin Developer Kit (Ampere architecture, compute capability 8.7, 64GB RAM)
NVIDIA Driver Version: 540.4.0
CUDA Version: 12.6.68
CUDNN Version: Not explicitly shown (not found via dpkg, presumably installed by default with TensorRT 10.3)

Operating System: Ubuntu 22.04.5 LTS (Jammy Jellyfish), Jetson Linux (aarch64 architecture, kernel 5.15.148-tegra)
Python Version: 3.10.17
PyTorch Version: 2.5.0
Baremetal: Yes (non-container environment)

Relevant Files

Model link: Local model coffee_machine_policy_simple.onnx (simplified ONNX model based on CNN architecture, containing convolution, batch normalization, ReLU, etc.)

Steps To Reproduce

Commands or scripts:

• Simplify the ONNX model:
  python -m onnxsim coffee_machine_policy.onnx coffee_machine_policy_simple.onnx
• Convert to TensorRT engine:
  trtexec --onnx=coffee_machine_policy_simple.onnx --saveEngine=coffee_machine_policy.engine --verbose

Full traceback of errors encountered:

plaintext
[09/04/2025-16:30:23] [E] Error[9]: Error Code: 9: Skipping tactic 0xf067e6205da31c2e due to exception initDeviceReservedSpace
[09/04/2025-16:30:24] [E] Error[9]: Error Code: 9: Skipping tactic 0x5deb29b7a8e275f7 due to exception initDeviceReservedSpace
[09/04/2025-16:30:24] [E] Error[9]: Error Code: 9: Skipping tactic 0xf64396b97c889179 due to exception initDeviceReservedSpace
[09/04/2025-16:30:24] [E] Error[9]: Error Code: 9: Skipping tactic 0xbdfdef6b84f7ccc9 due to exception initDeviceReservedSpace
[09/04/2025-16:30:24] [E] Error[9]: Error Code: 9: Skipping tactic 0x7bc32c782b800c48 due to exception initDeviceReservedSpace
[09/04/2025-16:30:24] [E] Error[10]: IBuilder::buildSerializedNetwork: Error Code 10: Internal Error (Could not find any implementation for node model/backbones.0/backbones.0.0/body/conv1/Conv + onnx::Mul_3979 + model/backbones.0/backbones.0.0/body/bn1/Mul + onnx::Add_3981 + model/backbones.0/backbones.0.0/body/bn1/Add + model/backbones.0/backbones.0.0/body/relu/Relu.)
[09/04/2025-16:30:24] [E] Error[1]: [checkMacros.cpp::catchCudaError::205] Error Code 1: Cuda Runtime (no kernel image is available for execution on the device)
[09/04/2025-16:30:24] [E] Engine could not be created from network
[09/04/2025-16:30:24] [E] Building engine failed
[09/04/2025-16:30:24] [E] Failed to create engine from model or file.
[09/04/2025-16:30:24] [E] Engine set up failed

Have you tried the latest release?: Yes, using the latest TensorRT 10.3 and PyTorch 2.5.0
Can this model run on other frameworks?: Yes, the ONNX model runs successfully on Jetson via ONNX Runtime (verified with polygraphy run coffee_machine_policy_simple.onnx --onnxrt)

Reproduce the failure with coffee_machine_policy_simple.onnx using the supplied onnxsim and trtexec commands on the Jetson AGX Orin environment. Start with the full trtexec log, especially the initDeviceReservedSpace and no-kernel-image errors, and compare the result with the successful ONNX Runtime run. Done means identifying a confirmed TensorRT-compatible configuration or a clearly documented cause and resolution.