pytorch / pytorch/TensorRT

❓ [Question] How do you build Torch-TensorRT from origin/main with dependence on tensorrt 8.5.2 from Jetpack5.1?

Open
#2,352 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

No Activity question Story: Build & Install & Packaging
Dominant language
Python
Stars
3k
Forks
410
Avg merge
3d 18h
Merged PRs (30d)
78

Description

❓ Question

When compiling the latest version of Torch-TensorRT from origin/main (2.2.0.dev0+76de80d0) on Jetpack5.1 using the latest locally compiled PyTorch (2.2.0a0+a683bc5) (so that I can use the latest v2 transforms in TorchVision (0.17.0a0+4cb3d80)), the resulting python package has a dependence on tensorrt version 8.6.1, but Jetpack5.1 only supports version 8.5.2.2-1+cuda11.4 and is thus not installable.
Is it possible to compile the latest Torch-TensorRT with dependence on the installed version of tensorrt?

Environment

Environment details 
br@nx:~/github/torch$ python /tmp/collect_env.py 
Collecting environment information...
PyTorch version: 2.2.0a0+a683bc5
Is debug build: False
CUDA used to build PyTorch: 11.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (aarch64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.27.5
Libc version: glibc-2.31

Python version: 3.8.10 (default, May 26 2023, 14:05:08)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.10.104-tegra-aarch64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: 11.4.315
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/aarch64-linux-gnu/libcudnn.so.8.6.0
/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8.6.0
/usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8.6.0
/usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8.6.0
/usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8.6.0
/usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8.6.0
/usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8.6.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          6
On-line CPU(s) list:             0-3
Off-line CPU(s) list:            4,5
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       2
Vendor ID:                       Nvidia
Model:                           0
Model name:                      ARMv8 Processor rev 0 (v8l)
Stepping:                        0x0
CPU max MHz:                     1907,2000
CPU min MHz:                     115,2000
BogoMIPS:                        62.50
L1d cache:                       256 KiB
L1i cache:                       512 KiB
L2 cache:                        4 MiB
L3 cache:                        4 MiB
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Branch predictor hardening
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm dcpop

Versions of relevant libraries:
[pip3] mypy==1.5.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.24.4
[pip3] numpy-quaternion==2022.4.3
[pip3] pytorch-ranger==0.1.1
[pip3] tensorrt==8.5.2.2
[pip3] torch==2.2.0a0+a683bc5
[pip3] torch-optimizer==0.3.0
[pip3] torchmetrics==0.11.3
[pip3] torchvision==0.17.0a0+4cb3d80
[conda] Could not collect

Torch and TorchVision are built with

export BUILD_TEST=OFF
export USE_FBGEMM=OFF  # Fails to build
export USE_NCCL=OFF    # Fails to build
export USE_KINETO=OFF  # Fails to build
export BUILD_SPLIT_CUDA=ON  # Required so that Torch-TensorRT finds the libraries it needs.
export _GLIBCXX_USE_CXX11_ABI=1  # Use the new C++ ABI

cd ~/github/torch/pytorch
python3 -m build -n
pip install dist/torch-<version>.whl

cd ~/github/torch/vision
python3 setup.py bdist_wheel  # Doesn't support the newer build module.
pip install dist/torchvision-<version>.whl
mkdir -p build; cd build
Torch_DIR=~/github/torch/pytorch/torch/share/cmake/Torch cmake -DCMAKE_BUILD_TYPE=Release -Wno-dev -DWITH_CUDA=on -GNinja -DCMAKE_INSTALL_PREFIX=~/.local ..
ninja install

WORKSPACE file used to build Torch-TensorRT on Jetpack5.1. Built with

cd ~/github/torch/Torch-TensorRT
bazel build //:libtorchtrt -c opt
sudo tar -xvzf bazel-bin/libtorchtrt.tar.gz -C /usr/local/
python3 setup.py bdist_wheel --use-cxx11-abi  # Doesn't support the newer build module.
pip install dist/torch_tensorrt-<version>.whl  # <-- fails to install due to tensorrt==8.6 dependency

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting setup.py and the linked WORKSPACE.txt used for the Jetpack5.1 build, then reproduce the bazel build and wheel installation with the listed PyTorch and TensorRT versions. Done means the Torch-TensorRT wheel builds and installs successfully against TensorRT 8.5.2.2 on Jetpack5.1.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
build-system, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.