Project-MONAI / Project-MONAI/model-zoo

VISTA-3D:About TensorRT speedup Error

Open
#703 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
338
Forks
106
PR merge metrics
No merged PRs in 30d

Description

When I use Vista3D, I encountered the following problems when running the command "python -m monai.bundle run --config_file "['configs/inference.json', 'configs/inference_trt.json']""

environment
TensorRT: 10.1.0
Torch-TensorRT Version: 2.4.0
Python version:3.10.15
CUDA version: 12.4
Torch Version:2.4.0+cu121
GPU:NVIDIA GeForce RTX 4090

error information
2024-10-24 10:30:17,210 - root - INFO - Restored all variables from .//models/model.pt
2024-10-24 10:30:17,211 - ignite.engine.engine.Vista3dEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs
2024-10-24 10:30:18,220 - INFO - Loading TensorRT engine: .//models/model.pt.image_encoder.encoder.plan
[I] Loading bytes from .//models/model.pt.image_encoder.encoder.plan
[E] IExecutionContext::enqueueV3: Error Code 1: Cask (Cask convolution execution)
2024-10-24 10:30:19,129 - INFO - Exception: CUDA ERROR: 700
Falling back to Pytorch ...
2024-10-24 10:30:19,131 - ignite.engine.engine.Vista3dEvaluator - ERROR - Current run is terminating due to exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

It looks like an environmental problem, but I don't know what went wrong.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the command with configs/inference.json and configs/inference_trt.json using the reported Python, PyTorch, CUDA, TensorRT, Torch-TensorRT, and GPU versions. Inspect the TensorRT engine loading step for model.pt.image_encoder.encoder.plan and capture whether the Cask or CUDA illegal-memory-access error consistently occurs. Done means identifying the environmental incompatibility or a documented configuration that avoids the failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.