Project-MONAI / Project-MONAI/model-zoo
VISTA-3D:About TensorRT speedup Error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 338
- Forks
- 106
- PR merge metrics
- No merged PRs in 30d
Description
When I use Vista3D, I encountered the following problems when running the command "python -m monai.bundle run --config_file "['configs/inference.json', 'configs/inference_trt.json']""
environment:
TensorRT: 10.1.0
Torch-TensorRT Version: 2.4.0
Python version:3.10.15
CUDA version: 12.4
Torch Version:2.4.0+cu121
GPU:NVIDIA GeForce RTX 4090
error information:
2024-10-24 10:30:17,210 - root - INFO - Restored all variables from .//models/model.pt
2024-10-24 10:30:17,211 - ignite.engine.engine.Vista3dEvaluator - INFO - Engine run resuming from iteration 0, epoch 0 until 1 epochs
2024-10-24 10:30:18,220 - INFO - Loading TensorRT engine: .//models/model.pt.image_encoder.encoder.plan
[I] Loading bytes from .//models/model.pt.image_encoder.encoder.plan
[E] IExecutionContext::enqueueV3: Error Code 1: Cask (Cask convolution execution)
2024-10-24 10:30:19,129 - INFO - Exception: CUDA ERROR: 700
Falling back to Pytorch ...
2024-10-24 10:30:19,131 - ignite.engine.engine.Vista3dEvaluator - ERROR - Current run is terminating due to exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
It looks like an environmental problem, but I don't know what went wrong.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the command with configs/inference.json and configs/inference_trt.json using the reported Python, PyTorch, CUDA, TensorRT, Torch-TensorRT, and GPU versions. Inspect the TensorRT engine loading step for model.pt.image_encoder.encoder.plan and capture whether the Cask or CUDA illegal-memory-access error consistently occurs. Done means identifying the environmental incompatibility or a documented configuration that avoids the failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100