NVIDIA / NVIDIA/TensorRT

Error Code 2: Internal Error (Assertion c != kNoColor failed. Topological sort failed, which means there's a circle in the graph.)

Open
#4,374 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

internal-bug-tracked Module:Engine Build triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

I export a variant of the YOLO model that contains a loop for post-processing abd ran into this error for topological sort. However, the model passes onnx_graphsurgeon's toposort(), nor could I spot any loop when visualizing it. A link to the ONNX file is attached, as well as a minimal python script to reproduce the error:

graphsurgeon toposort passed
[03/04/2025-06:21:50] [TRT] [I] [MemUsageChange] Init CUDA: CPU +19, GPU +0, now: CPU 42, GPU 102 (MiB)
[03/04/2025-06:21:52] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +963, GPU +194, now: CPU 1160, GPU 296 (MiB)
[03/04/2025-06:21:52] [TRT] [I] ----------------------------------------------------------------
[03/04/2025-06:21:52] [TRT] [I] Input filename: ./model.onnx
[03/04/2025-06:21:52] [TRT] [I] ONNX IR version: 0.0.9
[03/04/2025-06:21:52] [TRT] [I] Opset version: 19
[03/04/2025-06:21:52] [TRT] [I] Producer name: pytorch
[03/04/2025-06:21:52] [TRT] [I] Producer version: 2.4.0
[03/04/2025-06:21:52] [TRT] [I] Domain:
[03/04/2025-06:21:52] [TRT] [I] Model version: 0
[03/04/2025-06:21:52] [TRT] [I] Doc string:
[03/04/2025-06:21:52] [TRT] [I] ----------------------------------------------------------------
[03/04/2025-06:21:52] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[03/04/2025-06:21:52] [TRT] [E] [topSort.cpp::trivialChoice::325] Error Code 2: Internal Error (Assertion c != kNoColor failed. Topological sort failed, which means there's a circle in the graph.)

Environment

TensorRT Version: 10.7.0

NVIDIA GPU: T4

NVIDIA Driver Version: 550.90.07

CUDA Version: 12.4

CUDNN Version: N/A

Operating System: Debian

Python Version (if applicable):

Tensorflow Version (if applicable):

PyTorch Version (if applicable):

Baremetal or Container (if so, version):

Relevant Files

Model link:
https://drive.google.com/file/d/1rll4p_ejFeZWbhbKaO607awIIvQjleJC/view?usp=sharing

Python script:

import tensorrt as trt
import onnx_graphsurgeon as gs
import onnx

onnx_filename = "./model.onnx"
graph = gs.import_onnx(onnx.load(onnx_filename))
graph.toposort()
print("graphsurgeon toposort passed")

logger = trt.Logger(trt.Logger.INFO)
builder = trt.Builder(logger)
config = builder.create_builder_config()
network = builder.create_network()

# Read ONNX file
parser = trt.OnnxParser(network, logger)
parser.parse_from_file(onnx_filename)

# Add optimization profile
inputs = [network.get_input(i) for i in range(network.num_inputs)]
profile = builder.create_optimization_profile()
min_shape = (1, 3, 32, 32)
opt_shape = (8, 3, 384, 640)
max_shape = (8, 3, 640, 640)
for inp in inputs:
    profile.set_shape(inp.name, min=min_shape, opt=opt_shape, max=max_shape)
config.add_optimization_profile(profile)

# Fail here "Topological sort failed, which means there's a circle in the graph"
engine = builder.build_serialized_network(network, config)

Steps To Reproduce

Download the onnx file and run the python script above

Commands or scripts:
Have you tried the latest release?:

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided Python reproduction and model.onnx, then inspect the topSort.cpp::trivialChoice::325 location named in the error. Confirm whether the attached model triggers the failure in TensorRT 10.7.0 and compare it with the successful onnx_graphsurgeon toposort; done means the model builds without the topological-sort assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.