Error Code 2: Internal Error (Assertion c != kNoColor failed. Topological sort failed, which means there's a circle in the graph.)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Description
I export a variant of the YOLO model that contains a loop for post-processing abd ran into this error for topological sort. However, the model passes onnx_graphsurgeon's toposort(), nor could I spot any loop when visualizing it. A link to the ONNX file is attached, as well as a minimal python script to reproduce the error:
graphsurgeon toposort passed
[03/04/2025-06:21:50] [TRT] [I] [MemUsageChange] Init CUDA: CPU +19, GPU +0, now: CPU 42, GPU 102 (MiB)
[03/04/2025-06:21:52] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +963, GPU +194, now: CPU 1160, GPU 296 (MiB)
[03/04/2025-06:21:52] [TRT] [I] ----------------------------------------------------------------
[03/04/2025-06:21:52] [TRT] [I] Input filename: ./model.onnx
[03/04/2025-06:21:52] [TRT] [I] ONNX IR version: 0.0.9
[03/04/2025-06:21:52] [TRT] [I] Opset version: 19
[03/04/2025-06:21:52] [TRT] [I] Producer name: pytorch
[03/04/2025-06:21:52] [TRT] [I] Producer version: 2.4.0
[03/04/2025-06:21:52] [TRT] [I] Domain:
[03/04/2025-06:21:52] [TRT] [I] Model version: 0
[03/04/2025-06:21:52] [TRT] [I] Doc string:
[03/04/2025-06:21:52] [TRT] [I] ----------------------------------------------------------------
[03/04/2025-06:21:52] [TRT] [I] BuilderFlag::kTF32 is set but hardware does not support TF32. Disabling TF32.
[03/04/2025-06:21:52] [TRT] [E] [topSort.cpp::trivialChoice::325] Error Code 2: Internal Error (Assertion c != kNoColor failed. Topological sort failed, which means there's a circle in the graph.)
Environment
TensorRT Version: 10.7.0
NVIDIA GPU: T4
NVIDIA Driver Version: 550.90.07
CUDA Version: 12.4
CUDNN Version: N/A
Operating System: Debian
Python Version (if applicable):
Tensorflow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if so, version):
Relevant Files
Model link:
https://drive.google.com/file/d/1rll4p_ejFeZWbhbKaO607awIIvQjleJC/view?usp=sharing
Python script:
import tensorrt as trt
import onnx_graphsurgeon as gs
import onnx
onnx_filename = "./model.onnx"
graph = gs.import_onnx(onnx.load(onnx_filename))
graph.toposort()
print("graphsurgeon toposort passed")
logger = trt.Logger(trt.Logger.INFO)
builder = trt.Builder(logger)
config = builder.create_builder_config()
network = builder.create_network()
# Read ONNX file
parser = trt.OnnxParser(network, logger)
parser.parse_from_file(onnx_filename)
# Add optimization profile
inputs = [network.get_input(i) for i in range(network.num_inputs)]
profile = builder.create_optimization_profile()
min_shape = (1, 3, 32, 32)
opt_shape = (8, 3, 384, 640)
max_shape = (8, 3, 640, 640)
for inp in inputs:
profile.set_shape(inp.name, min=min_shape, opt=opt_shape, max=max_shape)
config.add_optimization_profile(profile)
# Fail here "Topological sort failed, which means there's a circle in the graph"
engine = builder.build_serialized_network(network, config)
Steps To Reproduce
Download the onnx file and run the python script above
Commands or scripts:
Have you tried the latest release?:
Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the provided Python reproduction and model.onnx, then inspect the topSort.cpp::trivialChoice::325 location named in the error. Confirm whether the attached model triggers the failure in TensorRT 10.7.0 and compare it with the successful onnx_graphsurgeon toposort; done means the model builds without the topological-sort assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100