pytorch / pytorch/TensorRT

❓ [Question] Jetson AGX Orin build and install torch_tensorrt wheel file Failed

Open
#3,331 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question Story: Build & Install & Packaging
Dominant language
Python
Stars
3k
Forks
410
Avg merge
3d 18h
Merged PRs (30d)
78

Description

❓ Question

I follow this tutorial to install Torch-TensorRT, but in the last step:

cuda_version=$(nvcc --version | grep Cuda | grep release | cut -d ',' -f 2 | sed -e 's/ release //g')
export TORCH_INSTALL_PATH=$(python -c "import torch, os; print(os.path.dirname(torch.__file__))")
export SITE_PACKAGE_PATH=${TORCH_INSTALL_PATH::-6}
export CUDA_HOME=/usr/local/cuda-${cuda_version}/
# replace the MODULE.bazel with the jetpack one
cat toolchains/jp_workspaces/MODULE.bazel.tmpl | envsubst > MODULE.bazel
# build and install torch_tensorrt wheel file
python setup.py --use-cxx11-abi install --user

some errors happened:

Run this command to start an interactive shell in an identical sandboxed environment:
(exec env - \
    LD_LIBRARY_PATH=/usr/lib/gcc/aarch64-linux-gnu/11:/usr/local/cuda-12.6/lib64: \
    PATH=/home/lab223/.cache/bazelisk/downloads/sha256/5a4cc979353671e438b9469b833924c2361e25a580cc278a75877aedc27c1c53/bin:/usr/lib/gcc/aarch64-linux-gnu/11:/home/lab223/anaconda3/envs/rnw/bin:/home/lab223/anaconda3/condabin:/usr/local/cuda-12.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/tmp \
  /home/lab223/.cache/bazel/_bazel_lab223/install/128438993754f9753a1e4f56fdd76124/linux-sandbox -t 15 -w /dev/shm -w /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/execroot/_main -w /tmp -M /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/_hermetic_tmp -m /tmp -S /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/stats.out -D /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/46/debug.out -- /bin/sh -i)
ERROR: /home/lab223/TensorRT/core/conversion/var/BUILD:20:11: Compiling core/conversion/var/Var.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/58/execroot/_main/bazel-out/aarch64-opt/bin/external/_main~_repo_rules~libtorch/_virtual_includes/ATen/ATen/core/DeprecatedTypePropertiesRegistry.h (???????)
ERROR: /home/lab223/TensorRT/core/conversion/converters/BUILD:59:11: Compiling core/conversion/converters/NodeConverterRegistry.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/57/execroot/_main/bazel-out/aarch64-opt/bin/external/_main~_repo_rules~libtorch/_virtual_includes/ATen/ATen/ops/cudnn_batch_norm_ops.h (???????)
ERROR: /home/lab223/TensorRT/core/conversion/converters/BUILD:39:11: Compiling core/conversion/converters/converter_util.cpp failed: I/O exception during sandboxed execution: /home/lab223/.cache/bazel/_bazel_lab223/3fb6c16c20f38dfc11e57e77e6eea473/sandbox/linux-sandbox/56/execroot/_main/external/_main~_repo_rules~libtorch/include/ATen/ops/native_dropout_backward_cpu_dispatch.h (???????)
Target //:libtorchtrt failed to build
INFO: Elapsed time: 1000.299s, Critical Path: 574.06s
INFO: 7984 processes: 7938 internal, 46 linux-sandbox.
ERROR: Build did NOT complete successfully

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

  • PyTorch Version (e.g., 1.0): 2.5.0
  • CPU Architecture: arm64(Jetson AGX Orin)
  • OS (e.g., Linux): Linux
  • How you installed PyTorch: pip
  • Build command you used (if compiling from source): python setup.py --use-cxx11-abi install --user
  • Are you using local sources or building from archives: building from archives
  • Python version: 3.10.15
  • CUDA version: 12.6
  • GPU models and configuration: -
  • Any other relevant information: Install torch_tensorrt in the model's anaconda virtual environment

Additional context

It seems a I/O exception.But Jetson still has 11GB of space.please help me!thanks!!!!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reported python setup.py --use-cxx11-abi install --user command and the generated MODULE.bazel based on toolchains/jp_workspaces/MODULE.bazel.tmpl. Inspect the Bazel failures in core/conversion/var/BUILD and core/conversion/converters/BUILD, then reproduce the build in the stated Jetson environment. Done means the libtorchtrt target builds and the wheel installs successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python, pytorch
Domain
build-system, embedded-iot, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.