How can I use the trex tool to analyze the engine generated by trtllm
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Description
I generated Phi-3-mini-4K-instruct engine based on ./example/phi/README.md and wanted to analyze it by trt-enging-explorer in trt. However, trex told me getPluginCreator could not find plugin: Gemmtensorrt_llm version: 1. I tried to add all of the .so files in ./tensorrt_llm/libs to trtexec's plugin library but it still didn't work.
The command line is as follows:
plugins=$(echo /app/tensorrt_llm/lib/*.so | sed 's/ / --plugins=/g')
plugins="--plugins=$plugins"
trtexec --loadEngine=./phi-engine/rank0.engine $plugins
Equivalent to:
trtexec --loadEngine=./phi-engine/rank0.engine --plugins=/app/tensorrt_llm/lib/libnvinfer_plugin_tensorrt_llm.so --plugins=/app/tensorrt_llm/lib/libtensorrt_llm.so --plugins=/app/tensorrt_llm/lib/libtensorrt_llm_nvrtc_wrapper.so --plugins=/app/tensorrt_llm/lib/libth_common.so
The error is as follows:
[E] [TRT] 3: getPluginCreator could not find plugin: GPTAttentiontensorrt_llm version: 1
Error[1]: [pluginV2Runner.cpp::load::294] Error Code 1: Serialization (Serialization assertion creator failed.Cannot deserialize plugin since corresponding IPluginCreator not found in Plugin Registry)
How can I add plugins from trtllm to trex?
Environment
TensorRT Version:10.0.1
NVIDIA GPU:RTX 4090
NVIDIA Driver Version: 535.183.01
CUDA Version:12.2
CUDNN Version:V12.4.99
Operating System:Ubuntu 22.04.4 LTS
Python Version (if applicable):3.10.12
Tensorflow Version (if applicable):
PyTorch Version (if applicable):2.3.0a0+40ec155e58.nv24.03
Baremetal or Container (if so, version):
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ./example/phi/README.md and the trtexec command shown in the issue, then inspect how TREX loads TensorRT engines and plugins. Reproduce the GPTAttention plugin deserialization error with rank0.engine and the listed .so files. Done means identifying a supported way to load the TensorRT-LLM plugins in TREX, or documenting the compatibility limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100