NVIDIA / NVIDIA/TensorRT

TensorRT engine hangs when calling get_binding_shape with PyTorch imported

Open
#4,608 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Module:ONNX Module:Runtime
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Description:
I encountered a problem when trying to visualize inference results of ONNX and TensorRT (TRT) face detection models on the same image. The program hangs and terminates when initializing the TensorRT engine.

Steps to reproduce:

  1. Load a TRT engine (face_detect_fp32.trt) using a shared CUDA context (shared_ctx).
  2. Import PyTorch (import torch) before loading the engine.
  3. In the TRTInference class, during initialization, the line:
shape = tuple(self.engine.get_binding_shape(i))

causes the program to hang.

  1. If I do not import PyTorch, everything works normally.

Code snippet (simplified):

import torch  # <- importing this causes the hang
import pycuda.driver as cuda
import tensorrt as trt

cuda.init()
ctx = cuda.Device(0).retain_primary_context()
ctx.push()

class TRTInference:
    def __init__(self, engine_path: str, ctx=None):
        with open(engine_path, "rb") as f, trt.Runtime(trt.Logger(trt.Logger.ERROR)) as runtime:
            self.engine = runtime.deserialize_cuda_engine(f.read())
        for i in range(self.engine.num_bindings):
            shape = tuple(self.engine.get_binding_shape(i))  # <--- hangs here

Observation:

  • The hang occurs only when importing PyTorch.
  • Using a shared CUDA context for TensorRT (ctx = cuda.Device(0).retain_primary_context()) does not prevent the issue.
  • I suspect there is a CUDA context conflict between PyTorch and PyCUDA/TensorRT.

Question:

  • Why does importing PyTorch cause get_binding_shape() to hang?
  • How can I modify the code to allow PyTorch and TensorRT to work in the same process safely?

Environment:

  • Python 3.8
  • PyTorch, TensorRT, PyCUDA
  • Shared CUDA context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the simplified TRTInference initializer and reproduce the hang with torch imported before PyCUDA, TensorRT, and get_binding_shape(). Compare the shared primary-context setup with the no-PyTorch case; done means the context interaction is identified and a safe same-process usage pattern is documented or verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.