TensorRT EP produces incorrect results with CUDA I/O Binding when input is populated asynchronously without an explicit stream dependency

### Describe the issue

We found a reproducible correctness issue when using ONNX Runtime TensorRT EP with CUDA I/O Binding.

The TensorRT engine itself is correct. The issue occurs when a CUDA-bound input buffer is populated asynchronously by external CUDA work and TensorRT EP consumes that buffer without an explicit CUDA stream happens-before relationship.

The same model and input produce correct results through:

• ORT CPU

• ORT CUDA

• native TensorRT 10.13.3.9

• an ORT-generated TensorRT engine executed directly with trtexec

• ORT TensorRT EP with host input

• ORT TensorRT EP with CUDA device input after explicit synchronization

The incorrect results occur only when the CUDA input buffer is populated asynchronously and inference starts without explicitly synchronizing the producer CUDA stream.

We initially suspected the D-FINE model, TF32, GridSample/deformable attention, TensorRT graph compilation, and TensorRT engine generation. Those possibilities were isolated and excluded.

Using the public D-FINE-M ONNX artifact from dfine-cpp v0.5.0, we observed the following over 32 images / 1,600 Top-50 detections:

For native TensorRT:

• Minimum IoU: 0.999568

• Mean IoU: 0.999995

For the final device-resident path after explicit synchronization:

• 1,600 / 1,600 matching detections

• Mean IoU: 0.999996984

• Maximum score difference: 0.000663111

An important observation was that copying the failing device input back to host immediately before inference also made the result correct. Since the D2H verification did not modify the input, this suggested a synchronization effect.

We then removed the D2H verification and explicitly synchronized the CUDA producer stream before Run(). This also restored full correctness.

Our current root-cause assessment is therefore:

External asynchronous CUDA work that populates a bound input buffer does not automatically establish a happens-before relationship with the TensorRT EP execution stream. IoBinding::SynchronizeInputs() did not provide the required ordering for this externally submitted CUDA work in our case.

This may be expected behavior, but the synchronization contract for externally populated CUDA-bound OrtValues appears easy to misunderstand.

We would appreciate clarification on:

1. Whether IoBinding::SynchronizeInputs() only synchronizes copies/work submitted internally by ONNX Runtime.

2. Whether applications must explicitly synchronize externally submitted CUDA work before Run().

3. Whether user_compute_stream, CUDA events, or another mechanism is the recommended way to establish this dependency without a full cudaStreamSynchronize().

4. Whether this external-stream requirement should be documented more explicitly for TensorRT EP + CUDA I/O Binding.

### To reproduce

A simplified reproduction pattern is:

// CUDA device input is already allocated and wrapped/bound as an OrtValue.

// Populate the input asynchronously using an external CUDA stream.
cudaMemcpyAsync(
device_input,
host_input,
input_bytes,
cudaMemcpyHostToDevice,
producer_stream);

// Bind CUDA input/output.
io_binding.BindInput(input_name, device_input_value);
io_binding.BindOutput(output_name, device_output_value);

// We also call the documented I/O Binding synchronization API.
io_binding.SynchronizeInputs();

// Run TensorRT EP.
session.Run(run_options, io_binding);

// In our environment this can produce numerically valid but incorrect results.
Adding explicit synchronization fixes the result:

cudaMemcpyAsync(
device_input,
host_input,
input_bytes,
cudaMemcpyHostToDevice,
producer_stream);

cudaStreamSynchronize(producer_stream);

io_binding.BindInput(input_name, device_input_value);
io_binding.BindOutput(output_name, device_output_value);

session.Run(run_options, io_binding);

// Correct result.
The same effect was observed when a D2H verification copy was inserted before Run(), which implicitly introduced sufficient synchronization.

Reproduction model:

dfine-cpp v0.5.0 public D-FINE-M ONNX artifact.

Validation conditions:

batch size: 1

input: 1x3x640x640

FP32

TF32 disabled

32 test images

Top-50 decoded detections per image

class-constrained one-to-one IoU comparison

The same ORT-generated TensorRT cached engine produces correct results when executed directly with native TensorRT / trtexec, which excludes TensorRT engine generation as the cause.

### Urgency

_No response_

### Platform

Windows

### OS Version

Windows NT 10.0, 64-bit

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

ONNX Runtime 1.30

### ONNX Runtime API

C++

### Architecture

X64

### Execution Provider

TensorRT

### Execution Provider Library Version

GPU: NVIDIA GeForce RTX 3060 CUDA: 13.x TensorRT: 10.13.3.9 Precision: FP32 TF32: disabled for final isolation tests

Start with the TensorRT EP CUDA I/O Binding path and reproduce the asynchronous producer case described in the issue, comparing it with explicit synchronization and host-input runs. Done means asynchronously populated CUDA inputs produce the same correct results without requiring an external synchronization workaround.

Written by the indexing model from the issue text.