refit_cuda_engine method is too slow

Description

I used the python tensorrt refitter class to load the LoRA weights of stable diffusion unet, but the refitter.refit_cuda_engine method is so slow, usually taking 4~5 seconds. Is there any way to improve the performance of refit_cuda_engine?

Environment

TensorRT Version:
8.6.1
NVIDIA GPU:
GeForce RTX 4090
NVIDIA Driver Version:
525.89.02
CUDA Version:
12.0
CUDNN Version:
8.9.2

Operating System:
Ubuntu 20.04.1
Python Version (if applicable):
3.9.16
Tensorflow Version (if applicable):

PyTorch Version (if applicable):
1.12.1
Baremetal or Container (if so, version):

Relevant Files

Model link:

Steps To Reproduce

Commands or scripts:

Have you tried the latest release?:

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):