trtexec: onnx to tensorRT convertion fails
Open
@brb-nv is already working on this.
Since May 24, 2024.
triaged
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Description
hi, i have an onnx model which i want to convert using trtexec:
[05/23/2024-21:39:30] [W] [TRT] onnx2trt_utils.cpp:514: Your ONNX model has been generated with double-typed weights, while TensorRT does not natively support double. Attempting to cast down to float.
[05/23/2024-21:39:30] [E] [TRT] onnx2trt_utils.cpp:748: Found unsupported datatype (8) when importing initializer: classes
[05/23/2024-21:39:30] [E] [TRT] ModelImporter.cpp:774: ERROR: ModelImporter.cpp:116 In function parseGraph:
[8] Assertion failed: convertOnnxWeights(initializer, &weights, ctx) && "Failed to import initializer."
it seems the onnx model is currently not supported with those datatypes - how to convert (what tool + settings) the model so tensorRT can use it?
Environment
TensorRT Version: 8.6.1
NVIDIA GPU: Nvidia T4
NVIDIA Driver Version:
CUDA Version: 12.x
CUDNN Version:
Operating System: Ubuntu 20.04
Python Version (if applicable):
Tensorflow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if so, version):
Relevant Files
Model link:
Steps To Reproduce
pip list:
pip list
Package Version
------------------ -------------
blinker 1.4
colored 2.2.4
coloredlogs 15.0.1
cryptography 3.4.8
dbus-python 1.2.18
distlib 0.3.8
distro 1.7.0
filelock 3.13.4
flatbuffers 24.3.25
httplib2 0.20.2
humanfriendly 10.0
importlib-metadata 4.6.4
jeepney 0.7.1
keyring 23.5.0
launchpadlib 1.10.16
lazr.restfulclient 0.14.4
lazr.uri 1.0.6
more-itertools 8.10.0
mpmath 1.3.0
numpy 1.26.4
oauthlib 3.2.0
onnxruntime-gpu 1.18.0
packaging 24.0
pip 24.0
platformdirs 4.2.0
polygraphy 0.49.9
protobuf 5.27.0
PyGObject 3.42.1
PyJWT 2.3.0
pyparsing 2.4.7
python-apt 2.4.0+ubuntu3
SecretStorage 3.3.1
setuptools 69.5.1
six 1.16.0
sympy 1.12
virtualenv 20.25.3
wadllib 1.3.6
wheel 0.43.0
zipp 1.0.0
Commands or scripts:
trtexec output:
/usr/src/tensorrt/bin/trtexec --onnx=model.onnx --saveEngine=model.plan
&&&& RUNNING TensorRT.trtexec [TensorRT v8603] # /usr/src/tensorrt/bin/trtexec --onnx=model.onnx --saveEngine=model.plan
[05/23/2024-21:39:22] [I] === Model Options ===
[05/23/2024-21:39:22] [I] Format: ONNX
[05/23/2024-21:39:22] [I] Model: model.onnx
[05/23/2024-21:39:22] [I] Output:
[05/23/2024-21:39:22] [I] === Build Options ===
[05/23/2024-21:39:22] [I] Max batch: explicit batch
[05/23/2024-21:39:22] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[05/23/2024-21:39:22] [I] minTiming: 1
[05/23/2024-21:39:22] [I] avgTiming: 8
[05/23/2024-21:39:22] [I] Precision: FP32
[05/23/2024-21:39:22] [I] LayerPrecisions:
[05/23/2024-21:39:22] [I] Layer Device Types:
[05/23/2024-21:39:22] [I] Calibration:
[05/23/2024-21:39:22] [I] Refit: Disabled
[05/23/2024-21:39:22] [I] Version Compatible: Disabled
[05/23/2024-21:39:22] [I] ONNX Native InstanceNorm: Disabled
[05/23/2024-21:39:22] [I] TensorRT runtime: full
[05/23/2024-21:39:22] [I] Lean DLL Path:
[05/23/2024-21:39:22] [I] Tempfile Controls: { in_memory: allow, temporary: allow }
[05/23/2024-21:39:22] [I] Exclude Lean Runtime: Disabled
[05/23/2024-21:39:22] [I] Sparsity: Disabled
[05/23/2024-21:39:22] [I] Safe mode: Disabled
[05/23/2024-21:39:22] [I] Build DLA standalone loadable: Disabled
[05/23/2024-21:39:22] [I] Allow GPU fallback for DLA: Disabled
[05/23/2024-21:39:22] [I] DirectIO mode: Disabled
[05/23/2024-21:39:22] [I] Restricted mode: Disabled
[05/23/2024-21:39:22] [I] Skip inference: Disabled
[05/23/2024-21:39:22] [I] Save engine: model.plan
[05/23/2024-21:39:22] [I] Load engine:
[05/23/2024-21:39:22] [I] Profiling verbosity: 0
[05/23/2024-21:39:22] [I] Tactic sources: Using default tactic sources
[05/23/2024-21:39:22] [I] timingCacheMode: local
[05/23/2024-21:39:22] [I] timingCacheFile:
[05/23/2024-21:39:22] [I] Heuristic: Disabled
[05/23/2024-21:39:22] [I] Preview Features: Use default preview flags.
[05/23/2024-21:39:22] [I] MaxAuxStreams: -1
[05/23/2024-21:39:22] [I] BuilderOptimizationLevel: -1
[05/23/2024-21:39:22] [I] Input(s)s format: fp32:CHW
[05/23/2024-21:39:22] [I] Output(s)s format: fp32:CHW
[05/23/2024-21:39:22] [I] Input build shapes: model
[05/23/2024-21:39:22] [I] Input calibration shapes: model
[05/23/2024-21:39:22] [I] === System Options ===
[05/23/2024-21:39:22] [I] Device: 0
[05/23/2024-21:39:22] [I] DLACore:
[05/23/2024-21:39:22] [I] Plugins:
[05/23/2024-21:39:22] [I] setPluginsToSerialize:
[05/23/2024-21:39:22] [I] dynamicPlugins:
[05/23/2024-21:39:22] [I] ignoreParsedPluginLibs: 0
[05/23/2024-21:39:22] [I]
[05/23/2024-21:39:22] [I] === Inference Options ===
[05/23/2024-21:39:22] [I] Batch: Explicit
[05/23/2024-21:39:22] [I] Input inference shapes: model
[05/23/2024-21:39:22] [I] Iterations: 10
[05/23/2024-21:39:22] [I] Duration: 3s (+ 200ms warm up)
[05/23/2024-21:39:22] [I] Sleep time: 0ms
[05/23/2024-21:39:22] [I] Idle time: 0ms
[05/23/2024-21:39:22] [I] Inference Streams: 1
[05/23/2024-21:39:22] [I] ExposeDMA: Disabled
[05/23/2024-21:39:22] [I] Data transfers: Enabled
[05/23/2024-21:39:22] [I] Spin-wait: Disabled
[05/23/2024-21:39:22] [I] Multithreading: Disabled
[05/23/2024-21:39:22] [I] CUDA Graph: Disabled
[05/23/2024-21:39:22] [I] Separate profiling: Disabled
[05/23/2024-21:39:22] [I] Time Deserialize: Disabled
[05/23/2024-21:39:22] [I] Time Refit: Disabled
[05/23/2024-21:39:22] [I] NVTX verbosity: 0
[05/23/2024-21:39:22] [I] Persistent Cache Ratio: 0
[05/23/2024-21:39:22] [I] Inputs:
[05/23/2024-21:39:22] [I] === Reporting Options ===
[05/23/2024-21:39:22] [I] Verbose: Disabled
[05/23/2024-21:39:22] [I] Averages: 10 inferences
[05/23/2024-21:39:22] [I] Percentiles: 90,95,99
[05/23/2024-21:39:22] [I] Dump refittable layers:Disabled
[05/23/2024-21:39:22] [I] Dump output: Disabled
[05/23/2024-21:39:22] [I] Profile: Disabled
[05/23/2024-21:39:22] [I] Export timing to JSON file:
[05/23/2024-21:39:22] [I] Export output to JSON file:
[05/23/2024-21:39:22] [I] Export profile to JSON file:
[05/23/2024-21:39:22] [I]
[05/23/2024-21:39:22] [I] === Device Information ===
[05/23/2024-21:39:22] [I] Selected Device: Tesla T4
[05/23/2024-21:39:22] [I] Compute Capability: 7.5
[05/23/2024-21:39:22] [I] SMs: 40
[05/23/2024-21:39:22] [I] Device Global Memory: 15102 MiB
[05/23/2024-21:39:22] [I] Shared Memory per SM: 64 KiB
[05/23/2024-21:39:22] [I] Memory Bus Width: 256 bits (ECC enabled)
[05/23/2024-21:39:22] [I] Application Compute Clock Rate: 1.59 GHz
[05/23/2024-21:39:22] [I] Application Memory Clock Rate: 5.001 GHz
[05/23/2024-21:39:22] [I]
[05/23/2024-21:39:22] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at.
[05/23/2024-21:39:22] [I]
[05/23/2024-21:39:22] [I] TensorRT version: 8.6.3
[05/23/2024-21:39:22] [I] Loading standard plugins
[05/23/2024-21:39:22] [I] [TRT] [MemUsageChange] Init CUDA: CPU +1, GPU +0, now: CPU 22, GPU 103 (MiB)
[05/23/2024-21:39:29] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +895, GPU +174, now: CPU 994, GPU 277 (MiB)
[05/23/2024-21:39:29] [I] Start parsing network model.
[05/23/2024-21:39:30] [I] [TRT] ----------------------------------------------------------------
[05/23/2024-21:39:30] [I] [TRT] Input filename: model.onnx
[05/23/2024-21:39:30] [I] [TRT] ONNX IR version: 0.0.7
[05/23/2024-21:39:30] [I] [TRT] Opset version: 12
[05/23/2024-21:39:30] [I] [TRT] Producer name: onnx.compose.merge_models
[05/23/2024-21:39:30] [I] [TRT] Producer version: 1.0
[05/23/2024-21:39:30] [I] [TRT] Domain:
[05/23/2024-21:39:30] [I] [TRT] Model version: 1
[05/23/2024-21:39:30] [I] [TRT] Doc string:
[05/23/2024-21:39:30] [I] [TRT] ----------------------------------------------------------------
[05/23/2024-21:39:30] [W] [TRT] onnx2trt_utils.cpp:514: Your ONNX model has been generated with double-typed weights, while TensorRT does not natively support double. Attempting to cast down to float.
[05/23/2024-21:39:30] [E] [TRT] onnx2trt_utils.cpp:748: Found unsupported datatype (8) when importing initializer: classes
[05/23/2024-21:39:30] [E] [TRT] ModelImporter.cpp:774: ERROR: ModelImporter.cpp:116 In function parseGraph:
[8] Assertion failed: convertOnnxWeights(initializer, &weights, ctx) && "Failed to import initializer."
[05/23/2024-21:39:30] [E] Failed to parse onnx file
[05/23/2024-21:39:30] [I] Finished parsing network model. Parse time: 0.583197
[05/23/2024-21:39:30] [E] Parsing model failed
[05/23/2024-21:39:30] [E] Failed to create engine from model or file.
[05/23/2024-21:39:30] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8603] # /usr/src/tensorrt/bin/trtexec --onnx=model.onnx --saveEngine=model.plan
trtexec output with verbose:
/usr/src/tensorrt/bin/trtexec --onnx=model.onnx --saveEngine=model.plan --verbose
[05/23/2024-22:14:38] [I] === Model Options ===
[05/23/2024-22:14:38] [I] Format: ONNX
[05/23/2024-22:14:38] [I] Model: model.onnx
[05/23/2024-22:14:38] [I] Output:
[05/23/2024-22:14:38] [I] === Build Options ===
[05/23/2024-22:14:38] [I] Max batch: explicit batch
[05/23/2024-22:14:38] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[05/23/2024-22:14:38] [I] minTiming: 1
[05/23/2024-22:14:38] [I] avgTiming: 8
[05/23/2024-22:14:38] [I] Precision: FP32
[05/23/2024-22:14:38] [I] LayerPrecisions:
[05/23/2024-22:14:38] [I] Layer Device Types:
[05/23/2024-22:14:38] [I] Calibration:
[05/23/2024-22:14:38] [I] Refit: Disabled
[05/23/2024-22:14:38] [I] Version Compatible: Disabled
[05/23/2024-22:14:38] [I] ONNX Native InstanceNorm: Disabled
[05/23/2024-22:14:38] [I] TensorRT runtime: full
[05/23/2024-22:14:38] [I] Lean DLL Path:
[05/23/2024-22:14:38] [I] Tempfile Controls: { in_memory: allow, temporary: allow }
[05/23/2024-22:14:38] [I] Exclude Lean Runtime: Disabled
[05/23/2024-22:14:38] [I] Sparsity: Disabled
[05/23/2024-22:14:38] [I] Safe mode: Disabled
[05/23/2024-22:14:38] [I] Build DLA standalone loadable: Disabled
[05/23/2024-22:14:38] [I] Allow GPU fallback for DLA: Disabled
[05/23/2024-22:14:38] [I] DirectIO mode: Disabled
[05/23/2024-22:14:38] [I] Restricted mode: Disabled
[05/23/2024-22:14:38] [I] Skip inference: Disabled
[05/23/2024-22:14:38] [I] Save engine: model.plan
[05/23/2024-22:14:38] [I] Load engine:
[05/23/2024-22:14:38] [I] Profiling verbosity: 0
[05/23/2024-22:14:38] [I] Tactic sources: Using default tactic sources
[05/23/2024-22:14:38] [I] timingCacheMode: local
[05/23/2024-22:14:38] [I] timingCacheFile:
[05/23/2024-22:14:38] [I] Heuristic: Disabled
[05/23/2024-22:14:38] [I] Preview Features: Use default preview flags.
[05/23/2024-22:14:38] [I] MaxAuxStreams: -1
[05/23/2024-22:14:38] [I] BuilderOptimizationLevel: -1
[05/23/2024-22:14:38] [I] Input(s)s format: fp32:CHW
[05/23/2024-22:14:38] [I] Output(s)s format: fp32:CHW
[05/23/2024-22:14:38] [I] Input build shapes: model
[05/23/2024-22:14:38] [I] Input calibration shapes: model
[05/23/2024-22:14:38] [I] === System Options ===
[05/23/2024-22:14:38] [I] Device: 0
[05/23/2024-22:14:38] [I] DLACore:
[05/23/2024-22:14:38] [I] Plugins:
[05/23/2024-22:14:38] [I] setPluginsToSerialize:
[05/23/2024-22:14:38] [I] dynamicPlugins:
[05/23/2024-22:14:38] [I] ignoreParsedPluginLibs: 0
[05/23/2024-22:14:38] [I]
[05/23/2024-22:14:38] [I] === Inference Options ===
[05/23/2024-22:14:38] [I] Batch: Explicit
[05/23/2024-22:14:38] [I] Input inference shapes: model
[05/23/2024-22:14:38] [I] Iterations: 10
[05/23/2024-22:14:38] [I] Duration: 3s (+ 200ms warm up)
[05/23/2024-22:14:38] [I] Sleep time: 0ms
[05/23/2024-22:14:38] [I] Idle time: 0ms
[05/23/2024-22:14:38] [I] Inference Streams: 1
[05/23/2024-22:14:38] [I] ExposeDMA: Disabled
[05/23/2024-22:14:38] [I] Data transfers: Enabled
[05/23/2024-22:14:38] [I] Spin-wait: Disabled
[05/23/2024-22:14:38] [I] Multithreading: Disabled
[05/23/2024-22:14:38] [I] CUDA Graph: Disabled
[05/23/2024-22:14:38] [I] Separate profiling: Disabled
[05/23/2024-22:14:38] [I] Time Deserialize: Disabled
[05/23/2024-22:14:38] [I] Time Refit: Disabled
[05/23/2024-22:14:38] [I] NVTX verbosity: 0
[05/23/2024-22:14:38] [I] Persistent Cache Ratio: 0
[05/23/2024-22:14:38] [I] Inputs:
[05/23/2024-22:14:38] [I] === Reporting Options ===
[05/23/2024-22:14:38] [I] Verbose: Enabled
[05/23/2024-22:14:38] [I] Averages: 10 inferences
[05/23/2024-22:14:38] [I] Percentiles: 90,95,99
[05/23/2024-22:14:38] [I] Dump refittable layers:Disabled
[05/23/2024-22:14:38] [I] Dump output: Disabled
[05/23/2024-22:14:38] [I] Profile: Disabled
[05/23/2024-22:14:38] [I] Export timing to JSON file:
[05/23/2024-22:14:38] [I] Export output to JSON file:
[05/23/2024-22:14:38] [I] Export profile to JSON file:
[05/23/2024-22:14:38] [I]
[05/23/2024-22:14:38] [I] === Device Information ===
[05/23/2024-22:14:38] [I] Selected Device: Tesla T4
[05/23/2024-22:14:38] [I] Compute Capability: 7.5
[05/23/2024-22:14:38] [I] SMs: 40
[05/23/2024-22:14:38] [I] Device Global Memory: 15102 MiB
[05/23/2024-22:14:38] [I] Shared Memory per SM: 64 KiB
[05/23/2024-22:14:38] [I] Memory Bus Width: 256 bits (ECC enabled)
[05/23/2024-22:14:38] [I] Application Compute Clock Rate: 1.59 GHz
[05/23/2024-22:14:38] [I] Application Memory Clock Rate: 5.001 GHz
[05/23/2024-22:14:38] [I]
[05/23/2024-22:14:38] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at.
[05/23/2024-22:14:38] [I]
[05/23/2024-22:14:38] [I] TensorRT version: 8.6.3
[05/23/2024-22:14:38] [I] Loading standard plugins
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::CoordConvAC version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::CropAndResizeDynamic version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::DecodeBbox3DPlugin version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::GenerateDetection_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::ModulatedDeformConv2d version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::NMSDynamic_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::PillarScatterPlugin version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::ProposalDynamic version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::Proposal version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::ROIAlign_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::ScatterND version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::Split version 1
[05/23/2024-22:14:38] [V] [TRT] Registered plugin creator - ::VoxelGeneratorPlugin version 1
[05/23/2024-22:14:38] [I] [TRT] [MemUsageChange] Init CUDA: CPU +1, GPU +0, now: CPU 22, GPU 103 (MiB)
[05/23/2024-22:14:38] [V] [TRT] Trying to load shared library libnvinfer_builder_resource.so.8.6.3
[05/23/2024-22:14:38] [V] [TRT] Loaded shared library libnvinfer_builder_resource.so.8.6.3
[05/23/2024-22:14:46] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +895, GPU +174, now: CPU 994, GPU 277 (MiB)
[05/23/2024-22:14:46] [V] [TRT] CUDA lazy loading is enabled.
[05/23/2024-22:14:46] [I] Start parsing network model.
[05/23/2024-22:14:46] [I] [TRT] ----------------------------------------------------------------
[05/23/2024-22:14:46] [I] [TRT] Input filename: model.onnx
[05/23/2024-22:14:46] [I] [TRT] ONNX IR version: 0.0.7
[05/23/2024-22:14:46] [I] [TRT] Opset version: 12
[05/23/2024-22:14:46] [I] [TRT] Producer name: onnx.compose.merge_models
[05/23/2024-22:14:46] [I] [TRT] Producer version: 1.0
[05/23/2024-22:14:46] [I] [TRT] Domain:
[05/23/2024-22:14:46] [I] [TRT] Model version: 1
[05/23/2024-22:14:46] [I] [TRT] Doc string:
[05/23/2024-22:14:46] [I] [TRT] ----------------------------------------------------------------
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::BatchedNMSDynamic_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::BatchedNMS_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::BatchTilePlugin_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::Clip_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::CoordConvAC version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::CropAndResizeDynamic version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::CropAndResize version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::DecodeBbox3DPlugin version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::DetectionLayer_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::EfficientNMS_Explicit_TF_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::EfficientNMS_Implicit_TF_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::EfficientNMS_ONNX_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::EfficientNMS_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::FlattenConcat_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::GenerateDetection_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::GridAnchor_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::GridAnchorRect_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::LReLU_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::ModulatedDeformConv2d version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::MultilevelCropAndResize_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::MultilevelProposeROI_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::MultiscaleDeformableAttnPlugin_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::NMSDynamic_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::NMS_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::Normalize_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::PillarScatterPlugin version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::PriorBox_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::ProposalDynamic version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::ProposalLayer_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::Proposal version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::PyramidROIAlign_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::Region_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::Reorg_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::ResizeNearest_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::ROIAlign_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::RPROI_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::ScatterND version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::SpecialSlice_TRT version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::Split version 1
[05/23/2024-22:14:46] [V] [TRT] Plugin creator already registered - ::VoxelGeneratorPlugin version 1
[05/23/2024-22:14:46] [V] [TRT] Adding network input: input_ids with dtype: int32, dimensions: (-1, -1)
[05/23/2024-22:14:46] [V] [TRT] Registering tensor: input_ids for ONNX tensor: input_ids
[05/23/2024-22:14:46] [V] [TRT] Adding network input: attention_mask with dtype: int32, dimensions: (-1, -1)
[05/23/2024-22:14:46] [V] [TRT] Registering tensor: attention_mask for ONNX tensor: attention_mask
[05/23/2024-22:14:46] [V] [TRT] Adding network input: token_type_ids with dtype: int32, dimensions: (-1, -1)
[05/23/2024-22:14:46] [V] [TRT] Registering tensor: token_type_ids for ONNX tensor: token_type_ids
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.embeddings.word_embeddings.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.embeddings.position_embeddings.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.embeddings.token_type_embeddings.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.embeddings.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.embeddings.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.0.attention.self.query.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.0.attention.self.key.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.0.attention.self.value.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.0.attention.output.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.0.attention.output.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.0.attention.output.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.0.intermediate.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.0.output.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.0.output.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.0.output.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.1.attention.self.query.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.1.attention.self.key.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.1.attention.self.value.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.1.attention.output.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.1.attention.output.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.1.attention.output.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.1.intermediate.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.1.output.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.1.output.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.1.output.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.2.attention.self.query.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.2.attention.self.key.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.2.attention.self.value.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.2.attention.output.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.2.attention.output.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.2.attention.output.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.2.intermediate.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.2.output.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.2.output.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.2.output.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.3.attention.self.query.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.3.attention.self.key.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.3.attention.self.value.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.3.attention.output.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.3.attention.output.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.3.attention.output.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.3.intermediate.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.3.output.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.3.output.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.3.output.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.4.attention.self.query.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.4.attention.self.key.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.4.attention.self.value.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.4.attention.output.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.4.attention.output.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.4.attention.output.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.4.intermediate.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.4.output.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.4.output.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.4.output.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.5.attention.self.query.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.5.attention.self.key.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.5.attention.self.value.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.5.attention.output.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.5.attention.output.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.5.attention.output.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.5.intermediate.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.5.output.dense.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.5.output.LayerNorm.weight
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: model_body.encoder.layer.5.output.LayerNorm.bias
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_822
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_823
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_826
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_832
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_833
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_834
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_835
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_836
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_839
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_845
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_846
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_847
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_848
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_849
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_852
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_858
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_859
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_860
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_861
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_862
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_865
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_871
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_872
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_873
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_874
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_875
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_878
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_884
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_885
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_886
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_887
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_888
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_891
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_897
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_898
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: onnx::MatMul_899
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: coef
[05/23/2024-22:14:46] [W] [TRT] onnx2trt_utils.cpp:514: Your ONNX model has been generated with double-typed weights, while TensorRT does not natively support double. Attempting to cast down to float.
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: intercept
[05/23/2024-22:14:46] [V] [TRT] Importing initializer: classes
[05/23/2024-22:14:46] [E] [TRT] onnx2trt_utils.cpp:748: Found unsupported datatype (8) when importing initializer: classes
[05/23/2024-22:14:46] [E] [TRT] ModelImporter.cpp:774: ERROR: ModelImporter.cpp:116 In function parseGraph:
[8] Assertion failed: convertOnnxWeights(initializer, &weights, ctx) && "Failed to import initializer."
[05/23/2024-22:14:46] [E] Failed to parse onnx file
[05/23/2024-22:14:46] [I] Finished parsing network model. Parse time: 0.229167
[05/23/2024-22:14:46] [E] Parsing model failed
[05/23/2024-22:14:46] [E] Failed to create engine from model or file.
[05/23/2024-22:14:46] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8603] # /usr/src/tensorrt/bin/trtexec --onnx=model.onnx --saveEngine=model.plan --verbose
Have you tried the latest release?:
why polygraphy wants to load a cuda 11.x library? i run this inside nvcr.io/nvidia/pytorch:24.03-py3 docker:
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):
polygraphy run model.onnx --onnxrt --providers "CUDAExecutionProvider"
[I] RUNNING | Command: /usr/local/bin/polygraphy run model.onnx --onnxrt --providers CUDAExecutionProvider
[I] onnxrt-runner-N0-05/24/24-00:42:23 | Activating and starting inference
[I] Creating ONNX-Runtime Inference Session with providers: ['CUDAExecutionProvider']
2024-05-24 00:42:23.653071659 [E:onnxruntime:Default, provider_bridge_ort.cc:1744 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory
2024-05-24 00:42:23.653106541 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:870 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirementsto ensure all dependencies are met.
[W] Input tensor: input_ids [shape=BoundedShape(['batch_size', 'sequence'], min=None, max=None)] | Will generate data of shape: [1, 1].
If this is incorrect, please provide a custom data loader.
[W] Input tensor: attention_mask [shape=BoundedShape(['batch_size', 'sequence'], min=None, max=None)] | Will generate data of shape: [1, 1].
If this is incorrect, please provide a custom data loader.
[W] Input tensor: token_type_ids [shape=BoundedShape(['batch_size', 'sequence'], min=None, max=None)] | Will generate data of shape: [1, 1].
If this is incorrect, please provide a custom data loader.
[I] onnxrt-runner-N0-05/24/24-00:42:23
---- Inference Input(s) ----
{input_ids [dtype=int64, shape=(1, 1)],
attention_mask [dtype=int64, shape=(1, 1)],
token_type_ids [dtype=int64, shape=(1, 1)]}
[W] Could not convert: object to a corresponding Polygraphy data type. Leaving this type in its source format.
[I] onnxrt-runner-N0-05/24/24-00:42:23
---- Inference Output(s) ----
{label [dtype=object, shape=(1,)],
probabilities [dtype=float64, shape=(1, 4)]}
[I] onnxrt-runner-N0-05/24/24-00:42:23 | Completed 1 iteration(s) in 3.36 ms | Average inference time: 3.36 ms.
[I] PASSED | Runtime: 0.404s | Command: /usr/local/bin/polygraphy run model.onnx --onnxrt --providers CUDAExecutionProvider
# pip list
Package Version
------------------ -------------
blinker 1.4
colored 2.2.4
coloredlogs 15.0.1
cryptography 3.4.8
dbus-python 1.2.18
distlib 0.3.8
distro 1.7.0
filelock 3.13.4
flatbuffers 24.3.25
httplib2 0.20.2
humanfriendly 10.0
importlib-metadata 4.6.4
jeepney 0.7.1
keyring 23.5.0
launchpadlib 1.10.16
lazr.restfulclient 0.14.4
lazr.uri 1.0.6
more-itertools 8.10.0
mpmath 1.3.0
numpy 1.26.4
oauthlib 3.2.0
onnxruntime-gpu 1.18.0
packaging 24.0
pip 24.0
platformdirs 4.2.0
polygraphy 0.49.9
protobuf 5.27.0
PyGObject 3.42.1
PyJWT 2.3.0
pyparsing 2.4.7
python-apt 2.4.0+ubuntu3
SecretStorage 3.3.1
setuptools 69.5.1
six 1.16.0
sympy 1.12
virtualenv 20.25.3
wadllib 1.3.6
wheel 0.43.0
zipp 1.0.0
cc @pranavm-nvidia @sachanub
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.