windows TensorRT inference time fluctuates greatly in some gpu drivers

Environment

TensorRT Version: 8.4.0.6 or 8.4.1.5
NVIDIA GPU: T600 or RTX 3060
NVIDIA Driver Version: 511.65 or 512.96
CUDA Version: 11.6 or 11.3
CUDNN Version: 8.x
Operating System: win10 or win 11

I have mentioned the same issue： https://github.com/NVIDIA/TensorRT/issues/1977#issue-1233391613

First, on the development computer, the 511.65 driver is pre-installed. TRT8.4.0, TRT 8.4.1, CUDA11.3, CUDA11.6 have been used. The two methods of counting time mentioned here have been used https://github.com/NVIDIA/TensorRT/issues/1977#issuecomment-1128539745. Inference times fluctuate wildly as we've seen here https://github.com/NVIDIA/TensorRT/issues/1977#issue-1233391613.

Second, On the development computer, I reduced the graphics driver to 473.47. The Inference time is very stable.

Third, On the new computer(win11, rtx 3060) where the product is deployed, the 512.96 drive is pre-installed. Inference times still fluctuate wildly.

What is the reason for this? Is the new driver more unstable?
After the product is released, the user's graphics card driver version cannot be controlled, how to solve it ?