Cuda Runtime (an illegal memory access was encountered) on calibrating Yolo_v5 model for int8 precision
Open
@akhilg-nv is already working on this.
Since Feb 11, 2025.
Module:Quantization
triaged
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Description
I am using this calibration script to generate the calib cache file for YoloV5 onnx model using EntropyCalibration2 method. But faced this issue after it starts calibrating for the 1st image.
Yolov5 model has topk and NMS nodes which has dynamic shape. How can we do calibration for such models?
Environment
TensorRT Version: 8.6.2.3
NVIDIA GPU: Orin Nano 8 GB
CUDA Version: 12.2
CUDNN Version: 8.9.4
Operating System: Jetpack 6.0
Python Version: 3.10
PyTorch Version: 2.3.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.