[Feature Request]

Describe the feature request

I am using onnxruntime version 1.14.0 with the programming language C#. When I try to use env.CreateAndRegisterAllocator(memInfo, arenaCfg), I encounter the following error: [ErrorCode:InvalidArgument] Only CPU devices are supported for now.

using (var memInfo = new OrtMemoryInfo(OrtMemoryInfo.allocatorCUDA, OrtAllocatorType.ArenaAllocator, 0, OrtMemType.Default))
using (var arenaCfg = new OrtArenaCfg(1024 * 1024 * 1024, 0, 512 * 1024 * 1024, 0))
{
      var env = OrtEnv.Instance();
      env.CreateAndRegisterAllocator(memInfo, arenaCfg);
 }
 

I would like to request for support for CUDA configuration in the onnxruntime library for C#. Alternatively, I would also like to request for a solution to clear the GPU memory after a model run without clearing the model itself.

Describe scenario use case

My scenario is that I need to use a lot of models on the GPU, and the memory is not enough. I want to clear the calculation cache after each model run, but for the overall time, I don't want to clear the model itself. I have already read - https://github.com/microsoft/onnxruntime/issues/9509#issuecomment-951546580 and - https://github.com/microsoft/onnxruntime/issues/13936; by upgrading to 1.14.0 I can already use runOp.AddRunConfigEntry("memory.enable_memory_arena_shrinkage", "gpu:0"), but now I cannot fully implement the functionality in C# due to the above feature not being supported.

It's also possible that my method is incorrect, so I sincerely request help. 😔🤞

Start with the C# APIs OrtEnv.CreateAndRegisterAllocator and RunOptions memory.enable_memory_arena_shrinkage, then read the guidance linked from issues 9509 and 13936. Done means establishing whether CUDA allocator configuration or post-run GPU memory clearing can be supported through C#, with the supported usage or limitation documented.