Failure of TensorRT 10.7 when running inference on A4500

Description

We successfully run inference with our model and observe some stability issues. After hours / days of runtime IExecutionContext:: enqueue(V2/V3) suddenly starts returning false and does not recover any more. This is followed by a driver crash visible in the Windows event log:

Display Driver nvlddmkm Stopped Responding.

The problem is that the documentation does not mention what this actually means, or how such situation (the false result) should be dealt with.

The errors happen randomly after within 2 to 200 hours of running the inference loop.

We are using TensorRT directly from C++ code.

What we've checked so far:

• There is no GPU resource leakage whatsoever as all resources (GPU RAM, streams etc.) are allocated before entering the inference loop, regular resource checks are performed.
• Input tensor dimensions are set before entering the inference loop so no dimensions mismatch can happen.
• We used CUDA Compute Sanitizer to confirm the inference loop is healthy.

Environment

TensorRT Version: 10.7

NVIDIA GPU: A4500 Ada (reproduced on Ampere as well).

NVIDIA Driver Version: Latest as of today (4.4.2025)

CUDA Version: 12.6

CUDNN Version: 9.6

Operating System: Windows Server 2022

Notable observations

When running on a Server with 2x A4500 Ada, despite otherwise independent and fed with different data, both inference loops (each GPU has its own loop with dedicated resources), stop working at the exact same time ending up with damaged inference engine instances, which, in my eyes, somehow points at the driver.

Are there any ways / strategies or whatever to narrow down the possible causes of these troubles? Nothing seems suspicious until the inference engine starts to return false from its execution methods and the driver crashes.

Thanks.

No repository file or test is named. Start with the IExecutionContext::enqueue(V2/V3) failure path and the reported Windows nvlddmkm crash, then review the TensorRT 10.7, CUDA 12.6, driver, and Windows Server 2022 details while reproducing the long-running inference loop. Done would require a confirmed cause or actionable guidance for handling the false return and driver failure.