INT8 calibration fails with trtexec

Description

A clear and concise description of the issue.

Environment

TensorRT Version: 8.5

NVIDIA GPU: Jetson Orin Nano

CUDA Version: 11.4

CUDNN Version: 11.4

Operating System:

Python Version (if applicable): 3.8.10

PyTorch Version (if applicable): '2.1.0a0+41361538.nv23.06'

I am using calibration code from here. And I have customized it to my application. I am using an Image Fusion model from here, that takes 2 images and fuse them to produce one image at the output. I have used the calibration data accordingly. However, the problem I can build the engine file with the example given here, and the engine file used for fusion but it return a dark image at the output. Poor fusion i think. When I try to build the engine using trtexec command, it gives error with calib cache file and --int8 option.

calibration code

import os
from glob import glob

import cv2
import numpy as np
import tensorrt as trt
from cuda import cudart

class Calibrator(trt.IInt8EntropyCalibrator):
    
    def __init__(self, calibaration_path, nCalibration, inputShape, cacheFile, homography):
        trt.IInt8EntropyCalibrator.__init__(self)

        # self.imageList = glob(calibaration_path + "*.jpg")[:100]
        self.vi = os.path.join(calibaration_path, "vi")
        self.ir = os.path.join(calibaration_path, "ir")
        self.imageList = os.listdir(self.vi)
        self.homography = homography
        if os.path.exists(self.homography):
            mat_data = np.load(self.homography)
            self.h_mat = mat_data["homography"]
        self.nCalibration = nCalibration
        self.shape = inputShape  # (N,C,H,W)
        self.buffeSize = trt.volume(inputShape) * trt.float32.itemsize
        self.cacheFile = cacheFile
        _, self.dIn = cudart.cudaMalloc(self.buffeSize)
        self.oneBatch = self.batchGenerator()

 

    def __del__(self):
        cudart.cudaFree(self.dIn)    
        
    def batchGenerator(self):
        for i in range(self.nCalibration):
            print("> calibration %d" % i)
            subImageList = np.random.choice(self.imageList, self.shape[0], replace=False)
            yield np.ascontiguousarray(self.loadImageList(subImageList))
    
    def perspective(self, img):
        if self.h_mat is not None:
            img = np.array(img)
            aligned_img = cv2.warpPerspective(img, self.h_mat, (img.shape[1], img.shape[0]))
            return aligned_img
        else:
            return None
    
    def loadImageList(self, imageList):
        res = np.empty(self.shape, dtype=np.float32)
        for i in range(self.shape[0]):
            vi_img = cv2.imread(os.path.join(self.vi, imageList[i]), cv2.IMREAD_GRAYSCALE).astype(np.float32)
            ir_img = cv2.imread(os.path.join(self.ir, imageList[i]), cv2.IMREAD_GRAYSCALE).astype(np.float32)
            
            ir_img = self.perspective(ir_img)
            
           
            vi_img = cv2.resize(vi_img, self.shape[2:])
            ir_img = cv2.resize(ir_img, self.shape[2:])
            
    
            vi_img = np.expand_dims(vi_img, axis=[0, 1])
            ir_img = np.expand_dims(ir_img, axis=[0, 1])
            img  = np.concatenate((ir_img, vi_img), axis=1)
            res[i] = img[0]
        return res
    
    def get_batch_size(self):  # necessary API
        return self.shape[0]
    
    def get_batch(self, nameList=None, inputNodeName=None):  # necessary API
        try:
            data = next(self.oneBatch)
            cudart.cudaMemcpy(self.dIn, data.ctypes.data, self.buffeSize, cudart.cudaMemcpyKind.cudaMemcpyHostToDevice)
            return [int(self.dIn)]
        except StopIteration:
                return None
    
    def read_calibration_cache(self):  # necessary API
        if os.path.exists(self.cacheFile):
            print("Succeed finding cahce file: %s" % (self.cacheFile))
            with open(self.cacheFile, "rb") as f:
                cache = f.read()
                return cache
        else:
            print("Failed finding int8 cache!")
            return
        
    def write_calibration_cache(self, cache):  # necessary API
        with open(self.cacheFile, "wb") as f:
            f.write(cache)
        print("Succeed saving int8 cache!")
        return



if __name__ == "__main__":
    cudart.cudaDeviceSynchronize()
    m = Calibrator("/home/jetson/Faizan/fusion/Saferail-AI/images", 5, (1, 2, 640, 640), "./fusion-int8.cache", "../camera_data/homography.npz")
    m.get_batch("FakeNameList")
    m.get_batch("FakeNameList")
    m.get_batch("FakeNameList")
    m.get_batch("FakeNameList")
    m.get_batch("FakeNameList")`

## code to generate engine
import os
from datetime import datetime as dt
from glob import glob
import argparse

import calibrate
import cv2
import numpy as np
import tensorrt as trt
import torch as t
import torch.nn.functional as F
from cuda import cudart
from torch.autograd import Variable


def read_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--onnx", default= "", type = str, help = "path to onnx file")
    parser.add_argument("--engine", default= "", type = str, help = "path to save engine file")
    parser.add_argument("--data", type = str, default="", help= 'paht to the data dir')
    parser.add_argument("--homography", type = str, default="", help= 'paht to the data dir')
    parser.add_argument("--cache", type = str, default="int8.cache", help= './int8.cache')
    parser.add_argument("--fp16", action= "store_true", help= 'use fp16')
    parser.add_argument("--int8", action= "store_true", help="use int8")
    opt = parser.parse_args()
    return opt
    

if __name__ == "__main__":
    # Read cmd args
    args = read_args()
    nCalibration = 1
    cudart.cudaDeviceSynchronize()
    
    # Set randon params
    np.random.seed(31193)
    t.manual_seed(97)
    t.cuda.manual_seed_all(97)
    t.backends.cudnn.deterministic = True


    # Values of some variables
    nHeight = 640
    nWidth = 640

    # Parse network, rebuild network and do inference in TensorRT
    logger = trt.Logger(trt.Logger.VERBOSE)
    builder = trt.Builder(logger)
    network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
    profile = builder.create_optimization_profile()
    config = builder.create_builder_config()
    if args.fp16:
        config.set_flag(trt.BuilderFlag.FP16)
    elif args.int8:
        config.set_flag(trt.BuilderFlag.INT8)
        config.int8_calibrator = calibrate.Calibrator(args.data, nCalibration, (1, 2, nHeight, nWidth), args.cache, args.homography)
        
    # Parse ONNX and build engine
    parser = trt.OnnxParser(network, logger)
    if not os.path.exists(args.onnx):
        print("Failed finding ONNX file!")
        exit()
    print("Succeeded finding ONNX file!")
    with open(args.onnx, "rb") as model:
        if not parser.parse(model.read()):
            print("Failed parsing .onnx file!")
            for error in range(parser.num_errors):
                print(parser.get_error(error))
            exit()
        print("Succeeded parsing .onnx file!")
    
    # Build engine
    inputTensor = network.get_input(0)
    profile.set_shape(inputTensor.name, [1, 2, nHeight, nWidth], [1, 2, nHeight, nWidth], [1, 2, nHeight, nWidth])
    config.add_optimization_profile(profile)

    # network.unmark_output(network.get_output(0))  # dont remove output
    output_tensor = network.get_output(0)
    engineString = builder.build_serialized_network(network, config)
    if engineString == None:
        print("Failed building engine!")
        exit()
    print("Succeeded building engine!")
    with open(args.engine, "wb") as f:
        f.write(engineString)
        
    # Run using TensorRT engine
    engine = trt.Runtime(logger).deserialize_cuda_engine(engineString)
    nIO = engine.num_io_tensors
    lTensorName = [engine.get_tensor_name(i) for i in range(nIO)]
    nInput = [engine.get_tensor_mode(lTensorName[i]) for i in range(nIO)].count(trt.TensorIOMode.INPUT)

    context = engine.create_execution_context()
    context.set_input_shape(lTensorName[0], [1, 2, nHeight, nWidth])
    for i in range(nIO):
        print("[%2d]%s->" % (i, "Input " if i < nInput else "Output"), engine.get_tensor_dtype(lTensorName[i]), engine.get_tensor_shape(lTensorName[i]), context.get_tensor_shape(lTensorName[i]), lTensorName[i])

    bufferH = []
    data = np.random.rand(1, 2, 640, 640)
    bufferH.append(np.ascontiguousarray(data))
    for i in range(nInput, nIO):
        bufferH.append(np.empty(context.get_tensor_shape(lTensorName[i]), dtype=trt.nptype(engine.get_tensor_dtype(lTensorName[i]))))
    bufferD = []
    for i in range(nIO):
        bufferD.append(cudart.cudaMalloc(bufferH[i].nbytes)[1])

    for i in range(nInput):
        cudart.cudaMemcpy(bufferD[i], bufferH[i].ctypes.data, bufferH[i].nbytes, cudart.cudaMemcpyKind.cudaMemcpyHostToDevice)

    for i in range(nIO):
        context.set_tensor_address(lTensorName[i], int(bufferD[i]))

    context.execute_async_v3(0)

    for i in range(nInput, nIO):
        cudart.cudaMemcpy(bufferH[i].ctypes.data, bufferD[i], bufferH[i].nbytes, cudart.cudaMemcpyKind.cudaMemcpyDeviceToHost)

    for i in range(nIO):
        print(lTensorName[i])
        print(bufferH[i])

    for b in bufferD:
        cudart.cudaFree(b)

    print("Succeeded running model in TensorRT!")`

the code gives poor results.

then I used trtexec method with the following commnad it gives error too.

trtexec --onnx=onnx_files/tardal.onnx --saveEngine=tensorrt_files/tardal-20aug-trtexec.trt --int8 --calib=int8_calibration/tardal_int8_new.cache --workspace=4096  --inputIOFormats=fp16:chw --outputIOFormats=fp16:chw

the error

08/20/2024-16:16:08] [I] [TRT] Starting Calibration.
[08/20/2024-16:16:08] [E] Error[1]: [executionContext.cpp::commonEmitDebugTensor::1652] Error Code 1: Cuda Runtime (invalid argument)
[08/20/2024-16:16:08] [E] Error[1]: [executionContext.cpp::executeInternal::966] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[08/20/2024-16:16:08] [E] Error[1]: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[08/20/2024-16:16:08] [E] Error[1]: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[08/20/2024-16:16:08] [E] Error[1]: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[08/20/2024-16:16:08] [E] Error[3]: [engine.cpp::~Engine::306] Error Code 3: API Usage Error (Parameter check failed at: runtime/api/engine.cpp::~Engine::306, condition: mObjectCounter.use_count() == 1. Destroying an engine object before destroying objects it created leads to undefined behavior.
)
[08/20/2024-16:16:08] [E] Error[1]: [resizingAllocator.cpp::deallocate::105] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
[08/20/2024-16:16:08] [E] Error[1]: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[08/20/2024-16:16:08] [E] Error[1]: [cudaDriverHelpers.cpp::operator()::30] Error Code 1: Cuda Driver (an illegal memory access was encountered)
[08/20/2024-16:16:08] [E] Error[2]: [calibrator.cpp::calibrateEngine::1181] Error Code 2: Internal Error (Assertion context->executeV2(&bindings[0]) failed. )
[08/20/2024-16:16:08] [E] Error[2]: [builder.cpp::buildSerializedNetwork::751] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[08/20/2024-16:16:08] [E] Engine could not be created from network
[08/20/2024-16:16:08] [E] Cuda failure: an illegal memory access was encountered
Aborted (core dumped)

Start by reproducing the failure with the provided trtexec command and TensorRT 8.5 environment, then inspect the supplied Python calibrator and calibration cache handling. Compare the custom engine-building path with trtexec and use the reported CUDA illegal-memory-access output to isolate the failing calibration step; done means INT8 calibration completes and the engine builds without the listed errors.