NVIDIA / NVIDIA/TensorRT

Failed to parse onnx file

Open
#4,587 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Module:ONNX
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

&&&& RUNNING TensorRT.trtexec [TensorRT v101303] [b9] # ./trtexec --onnx=ISTD_NS.onnx --saveEngine=ISTD_N.trt --iterations=10 --precisionConstraints=obey
[09/28/2025-09:21:51] [I] TF32 is enabled by default. Add --noTF32 flag to further improve accuracy with some performance cost.
[09/28/2025-09:21:51] [W] When --precisionConstraints flag is set to "obey" or "prefer", please add --layerPrecision/--layerOutputTypes flags to set layer-wise precisions and output types.
[09/28/2025-09:21:51] [I] === Model Options ===
[09/28/2025-09:21:51] [I] Format: ONNX
[09/28/2025-09:21:51] [I] Model: ISTD_NS.onnx
[09/28/2025-09:21:51] [I] Output:
[09/28/2025-09:21:51] [I] === Build Options ===
[09/28/2025-09:21:51] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default, tacticSharedMem: default
[09/28/2025-09:21:51] [I] avgTiming: 8
[09/28/2025-09:21:51] [I] Precision: FP32 (obey precision constraints)
[09/28/2025-09:21:51] [I] LayerPrecisions:
[09/28/2025-09:21:51] [I] Layer Device Types:
[09/28/2025-09:21:51] [I] Calibration:
[09/28/2025-09:21:51] [I] Refit: Disabled
[09/28/2025-09:21:51] [I] Strip weights: Disabled
[09/28/2025-09:21:51] [I] Version Compatible: Disabled
[09/28/2025-09:21:51] [I] ONNX Plugin InstanceNorm: Disabled
[09/28/2025-09:21:51] [I] ONNX kENABLE_UINT8_AND_ASYMMETRIC_QUANTIZATION_DLA flag: Disabled
[09/28/2025-09:21:51] [I] TensorRT runtime: full
[09/28/2025-09:21:51] [I] Lean DLL Path:
[09/28/2025-09:21:51] [I] Tempfile Controls: { in_memory: allow, temporary: allow }
[09/28/2025-09:21:51] [I] Exclude Lean Runtime: Disabled
[09/28/2025-09:21:51] [I] Sparsity: Disabled
[09/28/2025-09:21:51] [I] Safe mode: Disabled
[09/28/2025-09:21:51] [I] Build DLA standalone loadable: Disabled
[09/28/2025-09:21:51] [I] Allow GPU fallback for DLA: Disabled
[09/28/2025-09:21:51] [I] DirectIO mode: Disabled
[09/28/2025-09:21:51] [I] Restricted mode: Disabled
[09/28/2025-09:21:51] [I] Skip inference: Disabled
[09/28/2025-09:21:51] [I] Save engine: ISTD_N.trt
[09/28/2025-09:21:51] [I] Load engine:
[09/28/2025-09:21:51] [I] Profiling verbosity: 0
[09/28/2025-09:21:51] [I] Tactic sources: Using default tactic sources
[09/28/2025-09:21:51] [I] timingCacheMode: local
[09/28/2025-09:21:51] [I] timingCacheFile:
[09/28/2025-09:21:51] [I] Enable Compilation Cache: Enabled
[09/28/2025-09:21:51] [I] Enable Monitor Memory: Disabled
[09/28/2025-09:21:51] [I] errorOnTimingCacheMiss: Disabled
[09/28/2025-09:21:51] [I] Preview Features: Use default preview flags.
[09/28/2025-09:21:51] [I] MaxAuxStreams: -1
[09/28/2025-09:21:51] [I] BuilderOptimizationLevel: -1
[09/28/2025-09:21:51] [I] MaxTactics: -1
[09/28/2025-09:21:51] [I] Calibration Profile Index: 0
[09/28/2025-09:21:51] [I] Weight Streaming: Disabled
[09/28/2025-09:21:51] [I] Runtime Platform: Same As Build
[09/28/2025-09:21:51] [I] Debug Tensors:
[09/28/2025-09:21:51] [I] Distributive Independence: Disabled
[09/28/2025-09:21:51] [I] Mark Unfused Tensors As Debug Tensors: Disabled
[09/28/2025-09:21:51] [I] Input(s)s format: fp32:CHW
[09/28/2025-09:21:51] [I] Output(s)s format: fp32:CHW
[09/28/2025-09:21:51] [I] Input build shapes: model
[09/28/2025-09:21:51] [I] Input calibration shapes: model
[09/28/2025-09:21:51] [I] === System Options ===
[09/28/2025-09:21:51] [I] Device: 0
[09/28/2025-09:21:51] [I] DLACore:
[09/28/2025-09:21:51] [I] Plugins:
[09/28/2025-09:21:51] [I] setPluginsToSerialize:
[09/28/2025-09:21:51] [I] dynamicPlugins:
[09/28/2025-09:21:51] [I] ignoreParsedPluginLibs: 0
[09/28/2025-09:21:51] [I]
[09/28/2025-09:21:51] [I] === Inference Options ===
[09/28/2025-09:21:51] [I] Batch: Explicit
[09/28/2025-09:21:51] [I] Input inference shapes: model
[09/28/2025-09:21:51] [I] Iterations: 10
[09/28/2025-09:21:51] [I] Duration: 3s (+ 200ms warm up)
[09/28/2025-09:21:51] [I] Sleep time: 0ms
[09/28/2025-09:21:51] [I] Idle time: 0ms
[09/28/2025-09:21:51] [I] Inference Streams: 1
[09/28/2025-09:21:51] [I] ExposeDMA: Disabled
[09/28/2025-09:21:51] [I] Data transfers: Enabled
[09/28/2025-09:21:51] [I] Spin-wait: Disabled
[09/28/2025-09:21:51] [I] Multithreading: Disabled
[09/28/2025-09:21:51] [I] CUDA Graph: Disabled
[09/28/2025-09:21:51] [I] Separate profiling: Disabled
[09/28/2025-09:21:51] [I] Time Deserialize: Disabled
[09/28/2025-09:21:51] [I] Time Refit: Disabled
[09/28/2025-09:21:51] [I] NVTX verbosity: 0
[09/28/2025-09:21:51] [I] Persistent Cache Ratio: 0
[09/28/2025-09:21:51] [I] Optimization Profile Index: 0
[09/28/2025-09:21:51] [I] Weight Streaming Budget: 100.000000%
[09/28/2025-09:21:51] [I] Inputs:
[09/28/2025-09:21:51] [I] Debug Tensor Save Destinations:
[09/28/2025-09:21:51] [I] Dump All Debug Tensor in Formats:
[09/28/2025-09:21:51] [I] === Reporting Options ===
[09/28/2025-09:21:51] [I] Verbose: Disabled
[09/28/2025-09:21:51] [I] Averages: 10 inferences
[09/28/2025-09:21:51] [I] Percentiles: 90,95,99
[09/28/2025-09:21:51] [I] Dump refittable layers:Disabled
[09/28/2025-09:21:51] [I] Dump output: Disabled
[09/28/2025-09:21:51] [I] Profile: Disabled
[09/28/2025-09:21:51] [I] Export timing to JSON file:
[09/28/2025-09:21:51] [I] Export output to JSON file:
[09/28/2025-09:21:51] [I] Export profile to JSON file:
[09/28/2025-09:21:51] [I]
[09/28/2025-09:21:51] [I] === Device Information ===
[09/28/2025-09:21:51] [I] Available Devices:
[09/28/2025-09:21:51] [I] Device 0: "NVIDIA GeForce RTX 3080" UUID: GPU-b1617a48-0d9b-8c1c-a994-d9339659f7f4
[09/28/2025-09:21:51] [I] Selected Device: NVIDIA GeForce RTX 3080
[09/28/2025-09:21:51] [I] Selected Device ID: 0
[09/28/2025-09:21:51] [I] Selected Device UUID: GPU-b1617a48-0d9b-8c1c-a994-d9339659f7f4
[09/28/2025-09:21:51] [I] Compute Capability: 8.6
[09/28/2025-09:21:51] [I] SMs: 70
[09/28/2025-09:21:51] [I] Device Global Memory: 11901 MiB
[09/28/2025-09:21:51] [I] Shared Memory per SM: 100 KiB
[09/28/2025-09:21:51] [I] Memory Bus Width: 384 bits (ECC disabled)
[09/28/2025-09:21:51] [I] Application Compute Clock Rate: 1.785 GHz
[09/28/2025-09:21:51] [I] Application Memory Clock Rate: 9.501 GHz
[09/28/2025-09:21:51] [I]
[09/28/2025-09:21:51] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at.
[09/28/2025-09:21:51] [I]
[09/28/2025-09:21:51] [I] TensorRT version: 10.13.3
[09/28/2025-09:21:51] [I] Loading standard plugins
[09/28/2025-09:21:51] [I] [TRT] [MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 25, GPU 887 (MiB)
[09/28/2025-09:21:53] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +2134, GPU +396, now: CPU 2315, GPU 1283 (MiB)
[09/28/2025-09:21:53] [I] Start parsing network model.
[09/28/2025-09:21:53] [I] [TRT] ----------------------------------------------------------------
[09/28/2025-09:21:53] [I] [TRT] Input filename: ISTD_NS.onnx
[09/28/2025-09:21:53] [I] [TRT] ONNX IR version: 0.0.8
[09/28/2025-09:21:53] [I] [TRT] Opset version: 17
[09/28/2025-09:21:53] [I] [TRT] Producer name: pytorch
[09/28/2025-09:21:53] [I] [TRT] Producer version: 2.8.0
[09/28/2025-09:21:53] [I] [TRT] Domain:
[09/28/2025-09:21:53] [I] [TRT] Model version: 0
[09/28/2025-09:21:53] [I] [TRT] Doc string:
[09/28/2025-09:21:53] [I] [TRT] ----------------------------------------------------------------
[09/28/2025-09:21:53] [E] Error[3]: INetworkDefinition::addPoolingNd: Error Code 3: API Usage Error (Parameter check failed, condition: allDimsGtEq(windowSize, 1) && volume(windowSize) < (int64_t(std::pow(100000.0F, (nbSpatialDims) *0.5F))). )
[09/28/2025-09:21:53] [E] [TRT] ModelImporter.cpp:949: While parsing node number 1176 [AveragePool -> "/up_decoder2/coatt/AveragePool_output_0"]:
[09/28/2025-09:21:53] [E] [TRT] ModelImporter.cpp:950: --- Begin node ---
input: "/Add_1_output_0"
output: "/up_decoder2/coatt/AveragePool_output_0"
name: "/up_decoder2/coatt/AveragePool"
op_type: "AveragePool"
attribute {
name: "ceil_mode"
i: 0
type: INT
}
attribute {
name: "count_include_pad"
i: 1
type: INT
}
attribute {
name: "kernel_shape"
ints: 384
ints: 384
type: INTS
}
attribute {
name: "pads"
ints: 0
ints: 0
ints: 0
ints: 0
type: INTS
}
attribute {
name: "strides"
ints: 384
ints: 384
type: INTS
}

[09/28/2025-09:21:53] [E] [TRT] ModelImporter.cpp:951: --- End node ---
[09/28/2025-09:21:53] [E] [TRT] ModelImporter.cpp:954: ERROR: ModelImporter.cpp:177 In function parseNode:
[1] Exception occurred in - /up_decoder2/coatt/AveragePool
Internal Error!
[09/28/2025-09:21:53] [E] Failed to parse onnx file
[09/28/2025-09:21:53] [I] Finished parsing network model. Parse time: 0.0665538
[09/28/2025-09:21:53] [E] Parsing model failed
[09/28/2025-09:21:53] [E] Failed to create engine from model or file.
[09/28/2025-09:21:53] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v101303] [b9] # ./trtexec --onnx=ISTD_NS.onnx --saveEngine=ISTD_N.trt --iterations=10 --precisionConstraints=obey

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with ./trtexec --onnx=ISTD_NS.onnx and inspect node 1176, /up_decoder2/coatt/AveragePool, including its kernel_shape and strides. Trace the AveragePool parsing path from the trtexec entry point and compare the reported addPoolingNd validation with the ONNX model; done means the model parses and engine creation proceeds without this error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.