NVIDIA / NVIDIA/TensorRT

convert onnx to trt failed

Open
#1,882 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Module:ONNX triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

I've tried to convert onnx model to TRT model by trtexec but conversion failed. my model is segmentation model based on efficientnetb5.
these are the outputs:

trtexec --onnx=crack_onnx.onnx --saveEngine=crack.trt
&&&& RUNNING TensorRT.trtexec [TensorRT v8400] # F:\Libraries\TensorRT-8.4.0.6\bin\trtexec.exe --onnx=crack_onnx.onnx --saveEngine=crack.trt
[03/26/2022-11:00:28] [I] === Model Options ===
[03/26/2022-11:00:28] [I] Format: ONNX
[03/26/2022-11:00:28] [I] Model: crack_onnx.onnx
[03/26/2022-11:00:28] [I] Output:
[03/26/2022-11:00:28] [I] === Build Options ===
[03/26/2022-11:00:28] [I] Max batch: explicit batch
[03/26/2022-11:00:28] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[03/26/2022-11:00:28] [I] minTiming: 1
[03/26/2022-11:00:28] [I] avgTiming: 8
[03/26/2022-11:00:28] [I] Precision: FP32
[03/26/2022-11:00:28] [I] LayerPrecisions:
[03/26/2022-11:00:28] [I] Calibration:
[03/26/2022-11:00:28] [I] Refit: Disabled
[03/26/2022-11:00:28] [I] Sparsity: Disabled
[03/26/2022-11:00:28] [I] Safe mode: Disabled
[03/26/2022-11:00:28] [I] DirectIO mode: Disabled
[03/26/2022-11:00:28] [I] Restricted mode: Disabled
[03/26/2022-11:00:28] [I] Save engine: crack.trt
[03/26/2022-11:00:28] [I] Load engine:
[03/26/2022-11:00:28] [I] Profiling verbosity: 0
[03/26/2022-11:00:28] [I] Tactic sources: Using default tactic sources
[03/26/2022-11:00:28] [I] timingCacheMode: local
[03/26/2022-11:00:28] [I] timingCacheFile:
[03/26/2022-11:00:28] [I] Input(s)s format: fp32:CHW
[03/26/2022-11:00:28] [I] Output(s)s format: fp32:CHW
[03/26/2022-11:00:28] [I] Input build shapes: model
[03/26/2022-11:00:28] [I] Input calibration shapes: model
[03/26/2022-11:00:28] [I] === System Options ===
[03/26/2022-11:00:28] [I] Device: 0
[03/26/2022-11:00:28] [I] DLACore:
[03/26/2022-11:00:28] [I] Plugins:
[03/26/2022-11:00:28] [I] === Inference Options ===
[03/26/2022-11:00:28] [I] Batch: Explicit
[03/26/2022-11:00:28] [I] Input inference shapes: model
[03/26/2022-11:00:28] [I] Iterations: 10
[03/26/2022-11:00:28] [I] Duration: 3s (+ 200ms warm up)
[03/26/2022-11:00:28] [I] Sleep time: 0ms
[03/26/2022-11:00:28] [I] Idle time: 0ms
[03/26/2022-11:00:28] [I] Streams: 1
[03/26/2022-11:00:28] [I] ExposeDMA: Disabled
[03/26/2022-11:00:28] [I] Data transfers: Enabled
[03/26/2022-11:00:28] [I] Spin-wait: Disabled
[03/26/2022-11:00:28] [I] Multithreading: Disabled
[03/26/2022-11:00:28] [I] CUDA Graph: Disabled
[03/26/2022-11:00:28] [I] Separate profiling: Disabled
[03/26/2022-11:00:28] [I] Time Deserialize: Disabled
[03/26/2022-11:00:28] [I] Time Refit: Disabled
[03/26/2022-11:00:28] [I] Skip inference: Disabled
[03/26/2022-11:00:28] [I] Inputs:
[03/26/2022-11:00:28] [I] === Reporting Options ===
[03/26/2022-11:00:28] [I] Verbose: Disabled
[03/26/2022-11:00:28] [I] Averages: 10 inferences
[03/26/2022-11:00:28] [I] Percentile: 99
[03/26/2022-11:00:28] [I] Dump refittable layers:Disabled
[03/26/2022-11:00:28] [I] Dump output: Disabled
[03/26/2022-11:00:28] [I] Profile: Disabled
[03/26/2022-11:00:28] [I] Export timing to JSON file:
[03/26/2022-11:00:28] [I] Export output to JSON file:
[03/26/2022-11:00:28] [I] Export profile to JSON file:
[03/26/2022-11:00:28] [I]
[03/26/2022-11:00:28] [I] === Device Information ===
[03/26/2022-11:00:28] [I] Selected Device: NVIDIA GeForce GTX 960M
[03/26/2022-11:00:28] [I] Compute Capability: 5.0
[03/26/2022-11:00:28] [I] SMs: 5
[03/26/2022-11:00:28] [I] Compute Clock Rate: 1.176 GHz
[03/26/2022-11:00:28] [I] Device Global Memory: 4095 MiB
[03/26/2022-11:00:28] [I] Shared Memory per SM: 64 KiB
[03/26/2022-11:00:28] [I] Memory Bus Width: 128 bits (ECC disabled)
[03/26/2022-11:00:28] [I] Memory Clock Rate: 2.505 GHz
[03/26/2022-11:00:28] [I]
[03/26/2022-11:00:28] [I] TensorRT version: 8.4.0
[03/26/2022-11:00:29] [I] [TRT] [MemUsageChange] Init CUDA: CPU +270, GPU +0, now: CPU 8154, GPU 740 (MiB)
[03/26/2022-11:00:30] [I] [TRT] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 8304 MiB, GPU 740 MiB
[03/26/2022-11:00:30] [I] [TRT] [MemUsageSnapshot] End constructing builder kernel library: CPU 8304 MiB, GPU 741 MiB
[03/26/2022-11:00:30] [I] Start parsing network model
[03/26/2022-11:00:30] [I] [TRT] ----------------------------------------------------------------
[03/26/2022-11:00:30] [I] [TRT] Input filename:   crack_onnx.onnx
[03/26/2022-11:00:30] [I] [TRT] ONNX IR version:  0.0.4
[03/26/2022-11:00:30] [I] [TRT] Opset version:    9
[03/26/2022-11:00:30] [I] [TRT] Producer name:    tf2onnx
[03/26/2022-11:00:30] [I] [TRT] Producer version: 1.9.3
[03/26/2022-11:00:30] [I] [TRT] Domain:
[03/26/2022-11:00:30] [I] [TRT] Model version:    0
[03/26/2022-11:00:30] [I] [TRT] Doc string:
[03/26/2022-11:00:30] [I] [TRT] ----------------------------------------------------------------
[03/26/2022-11:00:30] [W] [TRT] onnx2trt_utils.cpp:365: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/26/2022-11:00:30] [E] Error[2]: [layers.cpp::nvinfer1::`anonymous-namespace'::equalIfKnown::177] Error Code 2: Internal Error (Assertion isConstant(y) failed. )
[03/26/2022-11:00:30] [E] [TRT] ModelImporter.cpp:748: While parsing node number 673 [Upsample -> "Upsample__2634:0"]:
[03/26/2022-11:00:30] [E] [TRT] ModelImporter.cpp:749: --- Begin node ---
[03/26/2022-11:00:30] [E] [TRT] ModelImporter.cpp:750: input: "StatefulPartitionedCall/model/decoder_stage0a_relu/Relu:0"
input: "Concat__2633:0"
output: "Upsample__2634:0"
name: "Upsample__2634"
op_type: "Upsample"
attribute {
  name: "mode"
  s: "nearest"
  type: STRING
}
domain: ""

[03/26/2022-11:00:30] [E] [TRT] ModelImporter.cpp:751: --- End node ---
[03/26/2022-11:00:30] [E] [TRT] ModelImporter.cpp:754: ERROR: builtin_op_importers.cpp:4655 In function importUpsample:
[8] Assertion failed: (outDims.nbDims == 1) && "The scales input must be 1D."
[03/26/2022-11:00:30] [E] Failed to parse onnx file
[03/26/2022-11:00:30] [I] Finish parsing network model
[03/26/2022-11:00:30] [E] Parsing model failed
[03/26/2022-11:00:30] [E] Failed to create engine from model.
[03/26/2022-11:00:30] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8400] # F:\Libraries\TensorRT-8.4.0.6\bin\trtexec.exe --onnx=crack_onnx.onnx --saveEngine=crack.trt
PS D:\MyWorks\Python\TensorRT> trtexec --onnx=crack.onnx --saveEngine=crack.trt
&&&& RUNNING TensorRT.trtexec [TensorRT v8400] # F:\Libraries\TensorRT-8.4.0.6\bin\trtexec.exe --onnx=crack.onnx --saveEngine=crack.trt
[03/26/2022-11:04:14] [I] === Model Options ===
[03/26/2022-11:04:14] [I] Format: ONNX
[03/26/2022-11:04:14] [I] Model: crack.onnx
[03/26/2022-11:04:14] [I] Output:
[03/26/2022-11:04:14] [I] === Build Options ===
[03/26/2022-11:04:14] [I] Max batch: explicit batch
[03/26/2022-11:04:14] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[03/26/2022-11:04:14] [I] minTiming: 1
[03/26/2022-11:04:14] [I] avgTiming: 8
[03/26/2022-11:04:14] [I] Precision: FP32
[03/26/2022-11:04:14] [I] LayerPrecisions:
[03/26/2022-11:04:14] [I] Calibration:
[03/26/2022-11:04:14] [I] Refit: Disabled
[03/26/2022-11:04:14] [I] Sparsity: Disabled
[03/26/2022-11:04:14] [I] Safe mode: Disabled
[03/26/2022-11:04:14] [I] DirectIO mode: Disabled
[03/26/2022-11:04:14] [I] Restricted mode: Disabled
[03/26/2022-11:04:14] [I] Save engine: crack.trt
[03/26/2022-11:04:14] [I] Load engine:
[03/26/2022-11:04:14] [I] Profiling verbosity: 0
[03/26/2022-11:04:14] [I] Tactic sources: Using default tactic sources
[03/26/2022-11:04:14] [I] timingCacheMode: local
[03/26/2022-11:04:14] [I] timingCacheFile:
[03/26/2022-11:04:14] [I] Input(s)s format: fp32:CHW
[03/26/2022-11:04:14] [I] Output(s)s format: fp32:CHW
[03/26/2022-11:04:15] [I] Input build shapes: model
[03/26/2022-11:04:15] [I] Input calibration shapes: model
[03/26/2022-11:04:15] [I] === System Options ===
[03/26/2022-11:04:15] [I] Device: 0
[03/26/2022-11:04:15] [I] DLACore:
[03/26/2022-11:04:15] [I] Plugins:
[03/26/2022-11:04:15] [I] === Inference Options ===
[03/26/2022-11:04:15] [I] Batch: Explicit
[03/26/2022-11:04:15] [I] Input inference shapes: model
[03/26/2022-11:04:15] [I] Iterations: 10
[03/26/2022-11:04:15] [I] Duration: 3s (+ 200ms warm up)
[03/26/2022-11:04:15] [I] Sleep time: 0ms
[03/26/2022-11:04:15] [I] Idle time: 0ms
[03/26/2022-11:04:15] [I] Streams: 1
[03/26/2022-11:04:15] [I] ExposeDMA: Disabled
[03/26/2022-11:04:15] [I] Data transfers: Enabled
[03/26/2022-11:04:15] [I] Spin-wait: Disabled
[03/26/2022-11:04:15] [I] Multithreading: Disabled
[03/26/2022-11:04:15] [I] CUDA Graph: Disabled
[03/26/2022-11:04:15] [I] Separate profiling: Disabled
[03/26/2022-11:04:15] [I] Time Deserialize: Disabled
[03/26/2022-11:04:15] [I] Time Refit: Disabled
[03/26/2022-11:04:15] [I] Skip inference: Disabled
[03/26/2022-11:04:15] [I] Inputs:
[03/26/2022-11:04:15] [I] === Reporting Options ===
[03/26/2022-11:04:15] [I] Verbose: Disabled
[03/26/2022-11:04:15] [I] Averages: 10 inferences
[03/26/2022-11:04:15] [I] Percentile: 99
[03/26/2022-11:04:15] [I] Dump refittable layers:Disabled
[03/26/2022-11:04:15] [I] Dump output: Disabled
[03/26/2022-11:04:15] [I] Profile: Disabled
[03/26/2022-11:04:16] [I] Export timing to JSON file:
[03/26/2022-11:04:16] [I] Export output to JSON file:
[03/26/2022-11:04:16] [I] Export profile to JSON file:
[03/26/2022-11:04:16] [I]
[03/26/2022-11:04:16] [I] === Device Information ===
[03/26/2022-11:04:16] [I] Selected Device: NVIDIA GeForce GTX 960M
[03/26/2022-11:04:16] [I] Compute Capability: 5.0
[03/26/2022-11:04:16] [I] SMs: 5
[03/26/2022-11:04:16] [I] Compute Clock Rate: 1.176 GHz
[03/26/2022-11:04:16] [I] Device Global Memory: 4095 MiB
[03/26/2022-11:04:16] [I] Shared Memory per SM: 64 KiB
[03/26/2022-11:04:16] [I] Memory Bus Width: 128 bits (ECC disabled)
[03/26/2022-11:04:16] [I] Memory Clock Rate: 2.505 GHz
[03/26/2022-11:04:16] [I]
[03/26/2022-11:04:16] [I] TensorRT version: 8.4.0
[03/26/2022-11:04:17] [I] [TRT] [MemUsageChange] Init CUDA: CPU +270, GPU +0, now: CPU 7692, GPU 740 (MiB)
[03/26/2022-11:04:18] [I] [TRT] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 7896 MiB, GPU 740 MiB
[03/26/2022-11:04:18] [I] [TRT] [MemUsageSnapshot] End constructing builder kernel library: CPU 7907 MiB, GPU 741 MiB
[03/26/2022-11:04:18] [I] Start parsing network model
[03/26/2022-11:04:18] [I] [TRT] ----------------------------------------------------------------
[03/26/2022-11:04:18] [I] [TRT] Input filename:   crack.onnx
[03/26/2022-11:04:18] [I] [TRT] ONNX IR version:  0.0.4
[03/26/2022-11:04:18] [I] [TRT] Opset version:    9
[03/26/2022-11:04:18] [I] [TRT] Producer name:    tf2onnx
[03/26/2022-11:04:18] [I] [TRT] Producer version: 1.9.3
[03/26/2022-11:04:18] [I] [TRT] Domain:
[03/26/2022-11:04:18] [I] [TRT] Model version:    0
[03/26/2022-11:04:18] [I] [TRT] Doc string:
[03/26/2022-11:04:18] [I] [TRT] ----------------------------------------------------------------
[03/26/2022-11:04:18] [W] [TRT] onnx2trt_utils.cpp:365: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[03/26/2022-11:04:19] [E] Error[2]: [layers.cpp::nvinfer1::`anonymous-namespace'::equalIfKnown::177] Error Code 2: Internal Error (Assertion isConstant(y) failed. )
[03/26/2022-11:04:19] [E] [TRT] ModelImporter.cpp:748: While parsing node number 673 [Upsample -> "Upsample__2634:0"]:
[03/26/2022-11:04:19] [E] [TRT] ModelImporter.cpp:749: --- Begin node ---
[03/26/2022-11:04:19] [E] [TRT] ModelImporter.cpp:750: input: "StatefulPartitionedCall/model/decoder_stage0a_relu/Relu:0"
input: "Concat__2633:0"
output: "Upsample__2634:0"
name: "Upsample__2634"
op_type: "Upsample"
attribute {
  name: "mode"
  s: "nearest"
  type: STRING
}
domain: ""

[03/26/2022-11:04:19] [E] [TRT] ModelImporter.cpp:751: --- End node ---
[03/26/2022-11:04:19] [E] [TRT] ModelImporter.cpp:754: ERROR: builtin_op_importers.cpp:4655 In function importUpsample:
[8] Assertion failed: (outDims.nbDims == 1) && "The scales input must be 1D."
[03/26/2022-11:04:19] [E] Failed to parse onnx file
[03/26/2022-11:04:19] [I] Finish parsing network model
[03/26/2022-11:04:19] [E] Parsing model failed
[03/26/2022-11:04:19] [E] Failed to create engine from model.
[03/26/2022-11:04:19] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8400] # F:\Libraries\TensorRT-8.4.0.6\bin\trtexec.exe --onnx=crack.onnx --saveEngine=crack.trt

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported trtexec command and the TensorRT parser output for ONNX node 673, named Upsample__2634, especially the scales input and the assertion in importUpsample. Reproduce the failure with the referenced ONNX model if available, then determine whether the parser can build the engine; done means a confirmed cause and either a successful conversion or a clearly documented limitation.

Written by the indexing model from the issue text.

Assessment

Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.