C# API: Stale results after first run when using TensorRT EP with trt_cuda_graph_enable

@chilo-ms is already working on this.

Since Jan 7, 2025.

I'm trying to run inference on a test model with TensorRT EP in C# with CUDA graph option enabled.
I couldn't find a canonical example in C#, so I tried to port the [official C++ example](https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#using-cuda-graphs-preview) and [this test](https://github.com/microsoft/onnxruntime/blob/4a196d15940b0f328735c888e2e861d67602ffcf/onnxruntime/test/shared_lib/test_inference.cc#L1867).

The C++ code [uses cudaMemcpy](https://github.com/microsoft/onnxruntime/blob/4a196d15940b0f328735c888e2e861d67602ffcf/onnxruntime/test/shared_lib/test_inference.cc#L1886) for shuffling the data around, but I hoped that C# API overtakes these low-level details.

According to the [comment here](https://github.com/microsoft/onnxruntime/blob/ef7f1ce08b58429b45f7e5b423dbc547353a6b8b/csharp/src/Microsoft.ML.OnnxRuntime/OrtValue.shared.cs#L611), I decided to create a memory chunk on CUDA device with `OrtValue.CreateAllocatedTensorValue`, and expected to be able to fill in the data using `GetTensorMutableDataAsSpan`.

However, in my tests the results didn't ever change after the first ("warm-up") run.

The problem can be reproduced with the following test.

I'm using a simple model, which accepts 2x3 float tensor, and returns the values multiplied by 2.

The C# code follows

```csharp
using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.Tensors;

using OrtTensorRTProviderOptions opt = new();
opt.UpdateOptions(new Dictionary()
{
{ "device_id", "0" },
{ "trt_cuda_graph_enable", "1" }
});

using SessionOptions sessionOptions = SessionOptions.MakeSessionOptionWithTensorrtProvider(opt);

using InferenceSession inferenceSession = new InferenceSession("dummy_model.onnx", sessionOptions);
using RunOptions runOptions = new RunOptions();

using OrtMemoryInfo cudaMemoryInfo = new OrtMemoryInfo(
OrtMemoryInfo.allocatorCUDA_PINNED, OrtAllocatorType.DeviceAllocator, 0, OrtMemType.Default);
// tried with ArenaAllocator as well
using OrtAllocator cudaAllocator = new OrtAllocator(inferenceSession, cudaMemoryInfo);

using OrtIoBinding ioBinding = inferenceSession.CreateIoBinding();

using OrtValue inputTensor = OrtValue.CreateAllocatedTensorValue(cudaAllocator, TensorElementType.Float, [2, 3]);
ioBinding.BindInput("x", inputTensor);

using OrtValue outputTensor = OrtValue.CreateAllocatedTensorValue(cudaAllocator, TensorElementType.Float, [2, 3]);
ioBinding.BindOutput("output_image", outputTensor);

// warm up run
Span inputSpan0 = inputTensor.GetTensorMutableDataAsSpan();
float[] inputData0 = [1, 2, 1, 2, 1, 2];
inputData0.AsSpan().CopyTo(inputSpan0);

ioBinding.SynchronizeBoundInputs();
inferenceSession.RunWithBinding(runOptions, ioBinding);

PrintTensor(outputTensor, "warm up result");

// actual run
float[] inputData1 = [3, 4, 4, 4, 4, 3];
Span inputSpan1 = inputTensor.GetTensorMutableDataAsSpan();
inputData1.CopyTo(inputSpan1);

ioBinding.SynchronizeBoundInputs();
inferenceSession.RunWithBinding(runOptions, ioBinding);
ioBinding.SynchronizeBoundOutputs();

PrintTensor(outputTensor, "actual run result");

void PrintTensor(OrtValue t, string header) where T : unmanaged =>
Console.WriteLine(header + ": " + string.Join(" ", t.GetTensorDataAsSpan().ToArray()));
```

However, I always get zeroes as output, whereas I expect `2 4 2 4 2 4` for warm-up run and `6 8 8 8 8 6` for actual run. If I add `ioBinding.BindInput("x", inputTensor);` into the warm-up run after copying the data, I get `2 4 2 4 2 4` output for both warm-up run and actual run, but never the expected `6 8 8 8 8 6` even if I add `ioBinding.BindInput("x", inputTensor);` again before actual run.

What is the canonical way to work with CUDA graph and TensorRT in C#? Is my code wrong, or is there a bug in C# TensorRT API?

---

Used versions:

* Nuget package Microsoft.ML.OnnxRuntime.Gpu: 1.19.2
* TensortRT: 10.2.0.19
* CUDA: 12.0.0 527.41 windows
* cuDNN: windows x86_64 9.5.0.50 cuda12

Platform: Windows 11