Building Tensorrt-LLM for Nvidia AGX with prescribed containers seems buggy and doesn't work.

System Info

CPU Architecture - arm64 (Nvdida AGX)
NVIDIA Driver - 540.4.0
CUDA version - 12.6

Running command sudo apt-cache show nvidia-jetpack

Package: nvidia-jetpack
Source: nvidia-jetpack (6.2)
Version: 6.2+b77
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 6.2+b77), nvidia-jetpack-dev (= 6.2+b77)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_6.2+b77_arm64.deb
Size: 29298
SHA256: 70553d4b5a802057f9436677ef8ce255db386fd3b5d24ff2c0a8ec0e485c59cd
SHA1: 9deab64d12eef0e788471e05856c84bf2a0cf6e6
MD5sum: 4db65dc36434fe1f84176843384aee23
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Package: nvidia-jetpack
Source: nvidia-jetpack (6.1)
Version: 6.1+b123
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 6.1+b123), nvidia-jetpack-dev (= 6.1+b123)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_6.1+b123_arm64.deb
Size: 29312
SHA256: b6475a6108aeabc5b16af7c102162b7c46c36361239fef6293535d05ee2c2929
SHA1: f0984a6272c8f3a70ae14cb2ca6716b8c1a09543
MD5sum: a167745e1d88a8d7597454c8003fa9a4
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I'm trying to build tensorrt-llm from source using what I assume is the official documentation. I'm building in a docker container based off nvcr.io/nvidia/pytorch:25.04-py3, which I think is one of the images we are supposed to use if we are building from source. This container already has the following:

tensorrt                   10.9.0.34
torch_tensorrt             2.7.0a0
pytorch-triton             3.2.0+git4b3bb1f8b.nvinternal
torch                      2.7.0a0+79aa17489c.nv25.4

I git cloned the release/0.19 and ran:

python3 ./scripts/build_wheel.py

Expected behavior

I expect to not be getting any errors if I'm simply following the documentation on building from source.

actual behavior

I got error:

  Using cached pyarrow_hotfix-0.7-py3-none-any.whl.metadata (3.6 kB)
INFO: pip is looking at multiple versions of datasets to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install -r requirements-dev.txt (line 3) because these package versions have conflicting dependencies.

The conflict is caused by:
    datasets 2.19.2 depends on dill<0.3.9 and >=0.3.0
    The user requested (constraint) dill==0.3.9

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

I check my dill version which is 0.39, I try to install dill version the within the correct range but pip cannot find any wheels within the range.
I basically commented out all packages which need dill and tried building again.

I get further but the build still ends in error:

......
-- Performing Test HAS_FLTO - Success
-- ========================= Importing and creating target nvinfer ==========================
-- Looking for library nvinfer
-- Library that was found /usr/lib/aarch64-linux-gnu/libnvinfer.so
-- ==========================================================================================
-- CUDAToolkit_VERSION 12.9 is greater or equal than 11.0, enable -DENABLE_BF16 flag
-- CUDAToolkit_VERSION 12.9 is greater or equal than 11.8, enable -DENABLE_FP8 flag
-- CUDAToolkit_VERSION 12.9 is greater or equal than 12.8, enable -DENABLE_FP4 flag
-- Found MPI_C: /opt/hpcx/ompi/lib/libmpi.so (found version "3.1")
-- Found MPI_CXX: /opt/hpcx/ompi/lib/libmpi.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
CMAKE_CUDA_FLAGS:  --expt-extended-lambda --expt-relaxed-constexpr --fatbin-options -compress-all
-- COMMON_HEADER_DIRS: /workspace/TensorRT-LLM/cpp
-- Found Python3: /usr/bin/python3.12 (found version "3.12.3") found components: Interpreter Development Development.Module Development.Embed
-- USE_CXX11_ABI is set by python Torch to 1
-- TORCH_CUDA_ARCH_LIST: 8.0;8.6;8.9;9.0a;10.0a;12.0a
CMake Warning at CMakeLists.txt:603 (message):
  Ignoring environment variable TORCH_CUDA_ARCH_LIST=8.0 8.6 9.0 10.0
  12.0+PTX


-- Found Python executable at /usr/bin/python3.12
-- Found Python libraries at /usr/lib/aarch64-linux-gnu
-- Found CUDA: /usr/local/cuda (found version "12.9") 
-- Found CUDAToolkit: /usr/local/cuda/include (found version "12.9.41")
-- PyTorch: CUDA detected: 12.9
-- PyTorch: CUDA nvcc is: /usr/local/cuda/bin/nvcc
-- PyTorch: CUDA toolkit directory: /usr/local/cuda
-- PyTorch: Header version is: 12.9
-- Found Python: /usr/bin/python3.12 (found version "3.12.3") found components: Interpreter
CMake Warning at /usr/local/lib/python3.12/dist-packages/torch/share/cmake/Caffe2/public/cuda.cmake:140 (message):
  Failed to compute shorthash for libnvrtc.so
Call Stack (most recent call first):
  /usr/local/lib/python3.12/dist-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:86 (include)
  /usr/local/lib/python3.12/dist-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:636 (find_package)


CMake Warning (dev) at /usr/local/lib/python3.12/dist-packages/cmake/data/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:441 (message):
  The package name passed to `find_package_handle_standard_args` (nvtx3) does
  not match the name of the calling package (Caffe2).  This can lead to
  problems in calling code that expects `find_package` result variables
  (e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
  /usr/local/lib/python3.12/dist-packages/torch/share/cmake/Caffe2/public/cuda.cmake:184 (find_package_handle_standard_args)
  /usr/local/lib/python3.12/dist-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:86 (include)
  /usr/local/lib/python3.12/dist-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:636 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found nvtx3: /usr/local/lib/python3.12/dist-packages/torch/include
-- USE_CUDNN is set to 0. Compiling without cuDNN support
-- USE_CUSPARSELT is set to 0. Compiling without cuSPARSELt support
-- USE_CUDSS is set to 0. Compiling without cuDSS support
-- USE_CUFILE is set to 0. Compiling without cuFile support
-- Added CUDA NVCC flags for: -gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_89,code=sm_89;-gencode;arch=compute_90a,code=sm_90a;-gencode;arch=compute_100a,code=sm_100a;-gencode;arch=compute_120a,code=sm_120a
CMake Warning at /usr/local/lib/python3.12/dist-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
  static library kineto_LIBRARY-NOTFOUND not found.
Call Stack (most recent call first):
  /usr/local/lib/python3.12/dist-packages/torch/share/cmake/Torch/TorchConfig.cmake:125 (append_torchlib_if_found)
  CMakeLists.txt:636 (find_package)


-- Found Torch: /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch.so
-- TORCH_CXX_FLAGS: -D_GLIBCXX_USE_CXX11_ABI=1
-- Found ucxx: /workspace/TensorRT-LLM/cpp/build/ucxx/build/ucxx-config.cmake (found version "0.42.0")
CMake Error at CMakeLists.txt:697 (file):
  file STRINGS file "/usr/local/tensorrt/include/NvInferVersion.h" cannot be
  read.


CMake Error at CMakeLists.txt:700 (string):
  string sub-command REGEX, mode MATCH needs at least 5 arguments total to
  command.


CMake Error at CMakeLists.txt:702 (string):
  string sub-command REGEX, mode MATCH needs at least 5 arguments total to
  command.


CMake Error at CMakeLists.txt:700 (string):
  string sub-command REGEX, mode MATCH needs at least 5 arguments total to
  command.


CMake Error at CMakeLists.txt:702 (string):
  string sub-command REGEX, mode MATCH needs at least 5 arguments total to
  command.


CMake Error at CMakeLists.txt:700 (string):
  string sub-command REGEX, mode MATCH needs at least 5 arguments total to
  command.


CMake Error at CMakeLists.txt:702 (string):
  string sub-command REGEX, mode MATCH needs at least 5 arguments total to
  command.


CMake Error at CMakeLists.txt:700 (string):
  string sub-command REGEX, mode MATCH needs at least 5 arguments total to
  command.


CMake Error at CMakeLists.txt:702 (string):
  string sub-command REGEX, mode MATCH needs at least 5 arguments total to
  command.


-- Building for TensorRT version: .., library version: 
CMake Error at CMakeLists.txt:716 (if):
  if given arguments:

    "LESS" "10"

  Unknown arguments specified


-- Configuring incomplete, errors occurred!
Traceback (most recent call last):
  File "/workspace/TensorRT-LLM/./scripts/build_wheel.py", line 551, in <module>
    main(**vars(args))
  File "/workspace/TensorRT-LLM/./scripts/build_wheel.py", line 234, in main
    build_run(cmake_configure_command)
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'cmake -DCMAKE_BUILD_TYPE="Release" -DBUILD_PYT="ON" -DBUILD_PYBIND="ON" -DNVTX_DISABLE="ON" -DBUILD_MICRO_BENCHMARKS=OFF -DBUILD_WHEEL_TARGETS="tensorrt_llm;nvinfer_plugin_tensorrt_llm;bindings;th_common;executorWorker"  -DTRT_LIB_DIR=/usr/local/tensorrt/targets/aarch64-linux-gnu/lib -DTRT_INCLUDE_DIR=/usr/local/tensorrt/include  -S "/workspace/TensorRT-LLM/cpp"' returned non-zero exit status 1.

additional notes

I tried to build with nvcr.io/nvidia/pytorch:25.01-py3 I get:

Collecting tensorrt_cu12==10.10.0.31 (from tensorrt~=10.10.0->-r /workspace/TensorRT-LLM/requirements.txt (line 24))
  Downloading tensorrt_cu12-10.10.0.31.tar.gz (18 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 35, in <module>
        File "/tmp/pip-install-mzi8006m/tensorrt-cu12_d412d616c1784449967845d7bfa3ddb4/setup.py", line 71, in <module>
          raise RuntimeError("TensorRT does not currently build wheels for Tegra systems")
      RuntimeError: TensorRT does not currently build wheels for Tegra systems
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Traceback (most recent call last):
  File "/workspace/TensorRT-LLM/./scripts/build_wheel.py", line 697, in <module>
    main(**vars(args))
  File "/workspace/TensorRT-LLM/./scripts/build_wheel.py", line 242, in main
    venv_python, venv_conan = setup_venv(project_dir,
                              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/TensorRT-LLM/./scripts/build_wheel.py", line 108, in setup_venv
    build_run(f'"{venv_python}" -m pip install -r "{requirements_file}"')
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '"/workspace/TensorRT-LLM/.venv-3.12/bin/python" -m pip install -r "/workspace/TensorRT-LLM/requirements-dev.txt"' returned non-zero exit status 1.

Start with scripts/build_wheel.py and requirements-dev.txt to trace the dependency setup, then inspect cpp/CMakeLists.txt around the TensorRT include and version checks. Reproduce with the documented nvcr.io/nvidia/pytorch container on the NVIDIA AGX environment; done means the prescribed build completes without the dill conflict or missing NvInferVersion.h errors.