Unexpected acceleration of diffusion-demo-like pipeline when running nsys profile on GPU A10

Description

I follow the diffusion-demo to accelerate my only SD pipeline, building VAE encoder, VAE decoder, Unet into different engines, while other models are torch. When not using nsys profile, my pipeline takes 616ms to complete, but when nsys profile is enabled, It only takes 600ms to complete, which is unexpected since nsys always slows down inference in my past experience. But when I export CUDA_LAUNCH_BLOCKING=1 to force device syncronize, nsys profile (688ms) now takes a little longer than not profiling (665ms) as usual. So I think the GPU syncronization behavior is somehow different when nsys profile is enabled.

BTW, I sync before timing, so the timer should be right. I also use cudart.cudaEventCreate to measure time, the result is the same: The Unet and VAE decoder takes less time to complete when using nsys profile.

I really want to now why, because finding the reason helps me to accelate the pipeline in futher degree. Does anyone encounter the same situtation?

Environment

TensorRT Version: tensorrt==9.2.0.post12.dev5

NVIDIA GPU: A10

NVIDIA Driver Version: 525.105.17

CUDA Version: V11.8

CUDNN Version:

Operating System:

Python Version (if applicable): 3.9.18

PyTorch Version (if applicable): 2.1.2