OOM error of TensorRT 10.12 when running engine build on GPU NVIDIA T4

Description

Hi,

My application build multiple TensorRT engines from ONNX models and they are run in a pipeline.
The problem I have is that I'm getting random OOM errors when building the first engine. I always build these engines sequentially. I deploy my application in a AWS cluster with g4dn.xlarge EC2 instance so I run this in NVIDIA T4 with 15gb which it is more than enough. My pipeline of models has a GPU memory peak of roughly 7GB during the engine build so there is enough memory.

This error happens randomly, maybe 1 out of 30 jobs fail with this error.

I run my application using docker with argument "--gpus all" and using base image nvidia/cuda:12.8.1-runtime-ubuntu22.04

Environment

TensorRT Version: 10.12

NVIDIA GPU: T4

NVIDIA Driver Version: 570.172.08

CUDA Version: 12.8.1

CUDNN Version: not used

Operating System: ubuntu 22

Relevant Files

Please find attached logs of the error:

trt_logs_jenkins.txt

Start by reviewing the attached trt_logs_jenkins.txt log and the reported environment: TensorRT 10.12, CUDA 12.8.1, Ubuntu 22.04, and an NVIDIA T4 in an AWS g4dn.xlarge instance. Reproduce the sequential engine builds in the Docker setup and determine why intermittent GPU out-of-memory errors occur; done means the cause is identified and a reliable correction or reproduction is documented.