microsoft / microsoft/onnxruntime

CUDA Plugin EP exits with 'Invalid memory type: -1' during inference (v1.25.1)

Open
#28,329 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

.NET api:CSharp ep:CUDA
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Description

When using the CUDA Execution Provider built as a plugin (`onnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON`), session creation succeeds but the first `session.Run()` crashes with:

```
terminate called after throwing an instance of 'onnxruntime::OnnxRuntimeException'
what(): /ort/include/onnxruntime/core/framework/ortdevice.h:77
constexpr OrtDevice::OrtDevice(DeviceType, MemoryType, VendorId, DeviceId, Alignment)
memory_type == MemType::DEFAULT || memory_type == MemType::HOST_ACCESSIBLE was false.
Invalid memory type: -1
```

The built-in CUDA EP (`AppendExecutionProvider_CUDA`) works correctly with the same model and inputs.

### Root Cause Analysis

PR #24985 refactored `OrtDevice::MemType` to only accept `DEFAULT(0)` and `HOST_ACCESSIBLE(5)`, adding a temporary validation check at `ortdevice.h:75-78`.

The **built-in CUDA EP** correctly maps legacy C API values via `CUDAExecutionProvider::GetOrtDeviceByMemType` (`cuda_execution_provider.cc:3264-3273`):
- `OrtMemTypeCPUOutput(-1)` → `OrtDevice(GPU, HOST_ACCESSIBLE, NVIDIA, id)` ✅

The **Plugin EP** factory creates an `OrtMemoryInfo` with `OrtMemTypeCPU(-1)` for pinned memory (`cuda_ep_factory.cc:206`). During inference, internal code reads the `mem_type` field from this `OrtMemoryInfo` and passes the raw `-1` value to an `OrtDevice` constructor instead of using the already-correct `.device` field — triggering the validation assert.

### Reproduction

- ORT v1.25.1 NuGet: `Microsoft.ML.OnnxRuntime.Gpu.Linux 1.25.1`
- Plugin .so: built from v1.25.1 tag with `onnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON`
- Platform: Ubuntu 24.04, CUDA 12.6.3, cuDNN 9.5.1, RTX 3070 (SM 86)
- C# API: `sessionOptions.AppendExecutionProvider(env, [device], options)`
- Model: Any ONNX model — crash occurs on first Run, not model-specific

Session creation via the Plugin EP API succeeds (all 6 GPUs register). The crash is during `session.Run()` or `session.RunWithBinding()`.

### Expected Behavior

The Plugin EP should map `OrtMemTypeCPUOutput(-1)` → `HOST_ACCESSIBLE(5)` at the plugin boundary, consistent with how the built-in EP handles it in `GetOrtDeviceByMemType`.

### Notes

- No C# tests exist for the Plugin EP path (`AppendExecutionProvider` with `OrtEpDevice`) in the ORT test suite — all CUDA C# tests use `AppendExecutionProvider_CUDA` (built-in path).
- The `ortdevice.h:75` comment says this check is "temporary to make sure we haven't missed any places where the deprecated values were used" — this appears to be one such place.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ortdevice.h:75-78, then inspect the plugin memory-info setup in cuda_ep_factory.cc:206 and compare it with CUDAExecutionProvider::GetOrtDeviceByMemType in cuda_execution_provider.cc:3264-3273. Reproduce the first session.Run() with the CUDA plugin and verify that the plugin path handles the legacy CPU output memory type without throwing; add regression coverage if the existing test structure supports it.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, csharp
Domain
backend-api-design, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.