NVIDIA / NVIDIA/TensorRT-Edge-LLM

Experimental OpenAI-compatible server fails to start with: ModuleNotFoundError: No module named 'tensorrt'

Open
#201 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
563
Forks
135
Avg merge
14h 13m
Merged PRs (30d)
1

Description

Describe the bug

The experimental OpenAI-compatible server currently fails to start with: ModuleNotFoundError: No module named 'tensorrt'.

Installing tensorrt 11.2.1.2 (using pip install tensorrt==11.2.1.2) does not resolve the issue. After installing tensorrt, starting the server instead fails with: FileNotFoundError: plugin library not found: build/libNvInfer_edgellm_plugin.so.

Expected behavior: The server should start successfully without requiring additional manual installation or configuration beyond the documented dependencies.

Actual behavior: The server fails to start both when TensorRT is not installed and after installing the specified TensorRT version, but with different errors.

Steps/Code to reproduce bug

Build configuration:
As described in TensorRT Edge-LLM on Jetson, I used an NVIDIA Jetson Thor as both the host and target device. I installed TensorRT Edge-LLM v0.10.1 by following the Installation Guide. The installation completed successfully without any issues.

$ python3 -m pip install pybind11==3.0.4
$ mkdir -p build
$ cd build
$ cmake .. \
    -DCMAKE_BUILD_TYPE=Release \
    -DTRT_PACKAGE_DIR=/usr \
    -DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake \
    -DEMBEDDED_TARGET=jetson-thor \
    -DCUDA_CTK_VERSION=13.0 \
    -DENABLE_CUTE_DSL=ALL \
    -DBUILD_PYTHON_BINDINGS=ON \
    -Dpybind11_DIR="$(python -m pybind11 --cmakedir)"
$ make -j$(nproc)

$ python3 -m pip install -e ".[server,server-tools]"

Runtime command used:

  • CLI:
tensorrt-edgellm-serve Qwen/Qwen3.5-0.8B \
  --max-input-len 4096 \
  --max-kv-cache-capacity 8192 \
  --port 8000
  • Python API:
from experimental.server import LLM, SamplingParams

llm = LLM(
    model="Qwen/Qwen3.5-0.8B",
    # cache_dir="/data/edgellm-cache",
    max_input_len=4096,
    max_kv_cache_capacity=8192,
)
result = llm.chat(
    [{"role": "user", "content": "Explain paged KV caches."}],
    SamplingParams(max_tokens=128, temperature=0),
)
print(result.text)

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 3
      1 from experimental.server import LLM, SamplingParams
      2 
----> 3 llm = LLM(
      4     model="Qwen/Qwen3.5-0.8B",
      5     # cache_dir="/data/edgellm-cache",
      6     max_input_len=4096,

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/server/runtime/engine.py:736, in LLM.__init__(self, model, cache_dir, engine_cache_max_size_gb, clear_engine_cache, max_input_len, max_batch_size, max_kv_cache_capacity, draft_top_k, draft_step, verify_tree_size, build_options, speculative_config, context_cache_config)
    733 if options.spec_type != "none":
    734     options = replace(options, tree_base=tree_base)
--> 736 prepared = prepare_model(
    737     model,
    738     cache_dir,
    739     options,
    740     max_cache_size_bytes=int(engine_cache_max_size_gb * (1 << 30)),
    741     clear_cache=clear_engine_cache,
    742 )
    743 self._cache_dir = cache_root(cache_dir)
    744 self._model_dir = prepared.model_dir

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/server/runtime/engine_build.py:439, in prepare_model(model, cache_dir, options, max_cache_size_bytes, clear_cache)
    433 build_options = replace(
    434     options,
    435     plugin_path=_resolve_plugin_path(options.plugin_path),
    436 )
    437 from experimental.builder.cli import main as builder_main
--> 439 builder_main(build_options.to_argv(model_dir, staging_dir))
    440 if not _is_ready(model_dir, staging_dir, build_options):
    441     raise RuntimeError(
    442         "the checkpoint-native builder did not produce a "
    443         f"complete runtime bundle for {model!r}")

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/builder/cli.py:347, in main(argv)
    342 args = parser.parse_args(argv)
    343 logging.basicConfig(
    344     level=logging.DEBUG if args.verbose else logging.INFO,
    345     format="%(asctime)s %(levelname)s %(name)s: %(message)s",
    346 )
--> 347 _build(args)

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/builder/cli.py:230, in _build(args)
    229 def _build(args: argparse.Namespace) -> None:
--> 230     from .core.builder import load_plugin_library
    232     plugin_path = args.plugin_path
    233     if plugin_path is None:

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/builder/core/builder.py:31
     28 from typing import Optional, Tuple
     30 import numpy as np
---> 31 import tensorrt as trt
     33 from ..ops.backend import Net
     34 from ..ops.functional.attention import KV_PAGE_SIZE

ModuleNotFoundError: No module named 'tensorrt'
  • Python API with tensorrt 11.2.1.2:
from experimental.server import LLM, SamplingParams

llm = LLM(
    model="Qwen/Qwen3.5-0.8B",
    # cache_dir="/data/edgellm-cache",
    max_input_len=4096,
    max_kv_cache_capacity=8192,
)
result = llm.chat(
    [{"role": "user", "content": "Explain paged KV caches."}],
    SamplingParams(max_tokens=128, temperature=0),
)
print(result.text)

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[1], line 3
      1 from experimental.server import LLM, SamplingParams
      2 
----> 3 llm = LLM(
      4     model="Qwen/Qwen3.5-0.8B",
      5     # cache_dir="/data/edgellm-cache",
      6     max_input_len=4096,

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/server/runtime/engine.py:736, in LLM.__init__(self, model, cache_dir, engine_cache_max_size_gb, clear_engine_cache, max_input_len, max_batch_size, max_kv_cache_capacity, draft_top_k, draft_step, verify_tree_size, build_options, speculative_config, context_cache_config)
    733 if options.spec_type != "none":
    734     options = replace(options, tree_base=tree_base)
--> 736 prepared = prepare_model(
    737     model,
    738     cache_dir,
    739     options,
    740     max_cache_size_bytes=int(engine_cache_max_size_gb * (1 << 30)),
    741     clear_cache=clear_engine_cache,
    742 )
    743 self._cache_dir = cache_root(cache_dir)
    744 self._model_dir = prepared.model_dir

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/server/runtime/engine_build.py:439, in prepare_model(model, cache_dir, options, max_cache_size_bytes, clear_cache)
    433 build_options = replace(
    434     options,
    435     plugin_path=_resolve_plugin_path(options.plugin_path),
    436 )
    437 from experimental.builder.cli import main as builder_main
--> 439 builder_main(build_options.to_argv(model_dir, staging_dir))
    440 if not _is_ready(model_dir, staging_dir, build_options):
    441     raise RuntimeError(
    442         "the checkpoint-native builder did not produce a "
    443         f"complete runtime bundle for {model!r}")

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/builder/cli.py:347, in main(argv)
    342 args = parser.parse_args(argv)
    343 logging.basicConfig(
    344     level=logging.DEBUG if args.verbose else logging.INFO,
    345     format="%(asctime)s %(levelname)s %(name)s: %(message)s",
    346 )
--> 347 _build(args)

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/builder/cli.py:240, in _build(args)
    238 args = _copy_args(args, plugin_path=plugin_path)
    239 bundle, components = _resolve_build_selection(args)
--> 240 plugin_handle = load_plugin_library(plugin_path)
    241 plan = _build_plan(args, bundle, components)
    242 LOGGER.info("Building %s engines for %s: %s", len(plan),
    243             bundle.root_model_type, ", ".join(label for label, *_ in plan))

File ~/aide/TensorRT-Edge-LLM/target/TensorRT-Edge-LLM/experimental/builder/core/builder.py:267, in load_plugin_library(plugin_path)
    265 path = plugin_path or "build/libNvInfer_edgellm_plugin.so"
    266 if not os.path.exists(path):
--> 267     raise FileNotFoundError(f"plugin library not found: {path}")
    268 handle = ctypes.CDLL(path, mode=ctypes.RTLD_GLOBAL)
    269 logger.info("Loaded plugin library %s", path)

FileNotFoundError: plugin library not found: build/libNvInfer_edgellm_plugin.so

System information

  • Platform: Nvidia Jetson Thor
  • Software release: JetPack 7.1 R38.4
  • CPU architecture: aarch64
  • GPU compute capability: SM110
    • Total device memory: 128 GB
  • Build typ: Release
  • Library versions:
    • TensorRT Edge-LLM version: 0.10.1
    • CUDA: 13.0
    • TensorRT: 10.13.3
    • C++ compiler: GCC 13.3.0
  • CMake options used:
    • CMAKE_TOOLCHAIN_FILE: cmake/aarch64_linux_toolchain.cmake
    • EMBEDDED_TARGET: jetson-thor
    • TRT_PACKAGE_DIR: /usr

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with experimental/server/runtime/engine.py and trace prepare_model into experimental/builder/cli.py and experimental/builder/core/builder.py, especially the TensorRT import and plugin-path resolution shown in the traceback. Reproduce the CLI or Python API failure on the documented Jetson Thor setup, then verify that the server starts with the documented dependencies and resolves the built plugin library without manual installation or configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, python
Domain
api, backend, build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.