NVIDIA / NVIDIA/TensorRT

failure of TensorRT 10.15 when running utils/process_engine.py on GPU A30

Open
#4,724 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Module:TREx
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

When I attempted to run this script tools/experimental/trt-engine-explorer/utils/process_engine.py for model performance analysis, an error occurred.

python utils/process_engine.py model.onnx ./output_dir fp16

error msg:

Profiling metadata: generated output file ./output_dir_opset13/model.onnx.engine.profile.metadata.json
Generating graph diagram: ./output_dir_opset13/model.onnx.engine.graph.json
Error: model.onnx.engine.graph.json.gv: syntax error in line 432 scanning a quoted string (missing endquote? longer than 16384?)
String starting:"LayerType:kgen\nMetadata:[ONNX Layer: ReduceSumSquare__2367]+[ONNX Layer: Div__2
Traceback (most recent call last):
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/env_trex/lib/python3.12/site-packages/graphviz/backend/execute.py", line 88, in run_check
    proc.check_returncode()
  File "/usr/lib/python3.12/subprocess.py", line 502, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '[PosixPath('dot'), '-Kdot', '-Tsvg', '-o', 'model.onnx.engine.graph.json.svg', 'model.onnx.engine.graph.json.gv']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/utils/process_engine.py", line 351, in <module>
    do_work(args)
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/utils/process_engine.py", line 345, in do_work
    process_engine(args, build, profile, draw)
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/utils/process_engine.py", line 279, in process_engine
    success = generate_engine_svg(args, add_suffix=build)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/utils/process_engine.py", line 250, in generate_engine_svg
    draw_engine(graph_json_fname, profiling_json_fname)
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/utils/draw_engine.py", line 179, in draw_engine
    trex.graphing.render_dot(graph, engine_json_fname, output_format)
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/trex/graphing.py", line 409, in render_dot
    dot_graph.render(outfile=output_fname, view=False, overwrite_source=False)
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/env_trex/lib/python3.12/site-packages/graphviz/_tools.py", line 185, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/env_trex/lib/python3.12/site-packages/graphviz/rendering.py", line 122, in render
    rendered = self._render(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/env_trex/lib/python3.12/site-packages/graphviz/_tools.py", line 185, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/env_trex/lib/python3.12/site-packages/graphviz/backend/rendering.py", line 326, in render
    execute.run_check(cmd,
  File "/data/handle_models/TensorRT/tools/experimental/trt-engine-explorer/env_trex/lib/python3.12/site-packages/graphviz/backend/execute.py", line 90, in run_check
    raise CalledProcessError(*e.args)
graphviz.backend.execute.CalledProcessError: Command '[PosixPath('dot'), '-Kdot', '-Tsvg', '-o', 'model.onnx.engine.graph.json.svg', 'model.onnx.engine.graph.json.gv']' returned non-zero exit status 1. [stderr: b'Error: model.onnx.engine.graph.json.gv: syntax error in line 432 scanning a quoted string (missing endquote? longer than 16384?)\nString starting:"LayerType:kgen\\nMetadata:[ONNX Layer: ReduceSumSquare__2367]+[ONNX Layer: Div__2\n']

Environment

TensorRT Version :10.15:

NVIDIA GPU : A30:

NVIDIA Driver Version : 550.144.03:

CUDA Version : 13.1:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with tools/experimental/trt-engine-explorer/utils/process_engine.py using the reported command and inspect the generated .gv file around line 432. Read utils/draw_engine.py and trex/graphing.py, especially render_dot, to trace how the graph label is emitted. Done means the reported model produces its SVG without a Graphviz quoted-string error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.