The same trt engine performs very differently in different programs.

Description

I conducted speed tests on a standard ResNet101 inference engine using two different programs.

The first program is one I created myself. I made some modifications to the code from https://github.com/NVIDIA-AI-IOT/Lidar_AI_Solution to adapt it to the input and output of ResNet101. The functionality of this program is very straightforward: it copies data from the host to the device, performs inference, and then transfers the results back to the host. The same task is repeated 1000 times in a loop on the same CUDA stream. To be clear, I didn’t modify the functional logic of the original project; I simply replaced the inference engine used in the original project with the ResNet101 engine.

The second program is the command-line tool trtexec. It also performs 1000 inferences, but the specific implementation details are handled internally by trtexec.

After the experiment, I encountered a puzzling phenomenon: the same engine takes an average of 4.7ms per inference in my program, while it only takes an average of 3.7ms per inference in trtexec.

I used the nsys tool to analyze the trtexec program and noticed that there is a cudaEventRecord CUDA API call before and after each inference execution. To ensure fair comparison, I inserted the same cudaEventRecord API calls at the corresponding points in my program.

From the timeline in nsys, it is apparent that trtexec achieves higher GPU utilization during the forward inference execution compared to my program. I believe this is the reason for the performance difference, but I still don’t know the root cause behind it.

I also noticed that trtexec has a parameter called maxAuxStreams. Initially, I thought this parameter might be affecting the fairness of the comparison, so I set it to 0. However, the results showed no significant changes.

I am stuck and would like to know how to make the profiling results of the two programs compatible with each other.

My program is like this. The trtexec command-line program is like this.

TensorRT Version:10.7.0.3

NVIDIA GPU:NVIDIA GeForce RTX 4070 Laptop

NVIDIA Driver Version:556.12

CUDA Version:12.5

CUDNN Version:8.4.0

Operating System:My machine runs on Windows, but since I use Docker containers to set up the environment, it should be WSL2

Python Version (if applicable):

Tensorflow Version (if applicable):

PyTorch Version (if applicable):

Baremetal or Container (if so, version):

Relevant Files

Model link:

Steps To Reproduce

Commands or scripts:

Have you tried the latest release?:

Can this model run on other frameworks? For example run ONNX model with ONNXRuntime (polygraphy run <model.onnx> --onnxrt):

Start with the reported comparison between the custom ResNet101 program and trtexec, using the nsys timeline and the cudaEventRecord calls as the first points of investigation. Review the TensorRT 10.7.0.3 setup on the RTX 4070 Laptop under WSL2, including the maxAuxStreams setting. Done means identifying the cause of the 4.7 ms versus 3.7 ms difference and documenting how to make the profiling results comparable.