microsoft / microsoft/onnxruntime
CUDA Plugin EP exits with 'Invalid memory type: -1' during inference (v1.25.1)
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Description
When using the CUDA Execution Provider built as a plugin (`onnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON`), session creation succeeds but the first `session.Run()` crashes with:
```
terminate called after throwing an instance of 'onnxruntime::OnnxRuntimeException'
what(): /ort/include/onnxruntime/core/framework/ortdevice.h:77
constexpr OrtDevice::OrtDevice(DeviceType, MemoryType, VendorId, DeviceId, Alignment)
memory_type == MemType::DEFAULT || memory_type == MemType::HOST_ACCESSIBLE was false.
Invalid memory type: -1
```
The built-in CUDA EP (`AppendExecutionProvider_CUDA`) works correctly with the same model and inputs.
### Root Cause Analysis
PR #24985 refactored `OrtDevice::MemType` to only accept `DEFAULT(0)` and `HOST_ACCESSIBLE(5)`, adding a temporary validation check at `ortdevice.h:75-78`.
The **built-in CUDA EP** correctly maps legacy C API values via `CUDAExecutionProvider::GetOrtDeviceByMemType` (`cuda_execution_provider.cc:3264-3273`):
- `OrtMemTypeCPUOutput(-1)` → `OrtDevice(GPU, HOST_ACCESSIBLE, NVIDIA, id)` ✅
The **Plugin EP** factory creates an `OrtMemoryInfo` with `OrtMemTypeCPU(-1)` for pinned memory (`cuda_ep_factory.cc:206`). During inference, internal code reads the `mem_type` field from this `OrtMemoryInfo` and passes the raw `-1` value to an `OrtDevice` constructor instead of using the already-correct `.device` field — triggering the validation assert.
### Reproduction
- ORT v1.25.1 NuGet: `Microsoft.ML.OnnxRuntime.Gpu.Linux 1.25.1`
- Plugin .so: built from v1.25.1 tag with `onnxruntime_BUILD_CUDA_EP_AS_PLUGIN=ON`
- Platform: Ubuntu 24.04, CUDA 12.6.3, cuDNN 9.5.1, RTX 3070 (SM 86)
- C# API: `sessionOptions.AppendExecutionProvider(env, [device], options)`
- Model: Any ONNX model — crash occurs on first Run, not model-specific
Session creation via the Plugin EP API succeeds (all 6 GPUs register). The crash is during `session.Run()` or `session.RunWithBinding()`.
### Expected Behavior
The Plugin EP should map `OrtMemTypeCPUOutput(-1)` → `HOST_ACCESSIBLE(5)` at the plugin boundary, consistent with how the built-in EP handles it in `GetOrtDeviceByMemType`.
### Notes
- No C# tests exist for the Plugin EP path (`AppendExecutionProvider` with `OrtEpDevice`) in the ORT test suite — all CUDA C# tests use `AppendExecutionProvider_CUDA` (built-in path).
- The `ortdevice.h:75` comment says this check is "temporary to make sure we haven't missed any places where the deprecated values were used" — this appears to be one such place.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ortdevice.h:75-78, then inspect the plugin memory-info setup in cuda_ep_factory.cc:206 and compare it with CUDAExecutionProvider::GetOrtDeviceByMemType in cuda_execution_provider.cc:3264-3273. Reproduce the first session.Run() with the CUDA plugin and verify that the plugin path handles the legacy CPU output memory type without throwing; add regression coverage if the existing test structure supports it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, csharp
- Domain
- backend-api-design, machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100