NVIDIA / NVIDIA/TensorRT

How can I use the trex tool to analyze the engine generated by trtllm

Open
#3,982 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Investigating Module:TREx triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description

I generated Phi-3-mini-4K-instruct engine based on ./example/phi/README.md and wanted to analyze it by trt-enging-explorer in trt. However, trex told me getPluginCreator could not find plugin: Gemmtensorrt_llm version: 1. I tried to add all of the .so files in ./tensorrt_llm/libs to trtexec's plugin library but it still didn't work.
The command line is as follows:

plugins=$(echo /app/tensorrt_llm/lib/*.so | sed 's/ / --plugins=/g')
plugins="--plugins=$plugins"
trtexec --loadEngine=./phi-engine/rank0.engine $plugins

Equivalent to:

trtexec --loadEngine=./phi-engine/rank0.engine --plugins=/app/tensorrt_llm/lib/libnvinfer_plugin_tensorrt_llm.so --plugins=/app/tensorrt_llm/lib/libtensorrt_llm.so --plugins=/app/tensorrt_llm/lib/libtensorrt_llm_nvrtc_wrapper.so --plugins=/app/tensorrt_llm/lib/libth_common.so

The error is as follows:

[E] [TRT] 3: getPluginCreator could not find plugin: GPTAttentiontensorrt_llm version: 1
Error[1]: [pluginV2Runner.cpp::load::294] Error Code 1: Serialization (Serialization assertion creator failed.Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)

How can I add plugins from trtllm to trex?

Environment

TensorRT Version:10.0.1

NVIDIA GPU:RTX 4090

NVIDIA Driver Version: 535.183.01

CUDA Version:12.2

CUDNN Version:V12.4.99

Operating System:Ubuntu 22.04.4 LTS

Python Version (if applicable):3.10.12

Tensorflow Version (if applicable):

PyTorch Version (if applicable):2.3.0a0+40ec155e58.nv24.03

Baremetal or Container (if so, version):

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ./example/phi/README.md and the trtexec command shown in the issue, then inspect how TREX loads TensorRT engines and plugins. Reproduce the GPTAttention plugin deserialization error with rank0.engine and the listed .so files. Done means identifying a supported way to load the TensorRT-LLM plugins in TREX, or documenting the compatibility limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.