microsoft / microsoft/onnxruntime

WebGPU EP: concurrent Session.Run() across two sessions segfaults (Linux/Vulkan, same or different GPU)

Open
#32,561 2 comments 0 reactions 1 assignee Claimed by @xiaofeihan1 View on GitHub
.NET api:CSharp ep:WebGPU platform:web
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

Calling `Session.Run()` concurrently (from two separate threads, at the same time) on two different `InferenceSession` instances that both use the native **WebGPU plugin execution provider** (`Microsoft.ML.OnnxRuntime.EP.WebGpu`) segfaults the process (SIGSEGV / exit code 139), on **Linux via the Vulkan backend** (Dawn).

This reproduces:
- with the two sessions pinned to **two different physical GPUs** (an NVIDIA discrete GPU and an Intel integrated GPU, both enumerated via `OrtEnv.GetEpDevices()`), and
- with two sessions pinned to the **same** `OrtEpDevice`.

In both cases:
- Constructing the two sessions **concurrently** is fine (no crash).
- Running the two sessions' `Run()` calls **sequentially** (one completes before the other starts) is fine (no crash), even after concurrent construction.
- Only **concurrent `Run()`** calls crash.

This looks like the same underlying issue as #27592 / #31627 ("WebGPU EP crashes on macOS when running concurrent multi threaded inference," still open, last confirmed on ORT 1.28), but reproduced on a different Dawn backend (Vulkan/Linux instead of Metal/macOS) and additionally across two separate physical GPUs, which that issue doesn't appear to cover. Filing separately since the backend/platform and cross-GPU angle are new data points — happy to have this folded into #31627 instead if the maintainers prefer.

It's worth flagging as more than a minor edge case because **it's silent** — no exception, no ORT-level error, just a native crash — and nothing in the WebGPU EP's documentation states that concurrent `Run()` across sessions is unsafe (in contrast to the well-known guidance that a *single* session's `Run()` is safe to call concurrently, which holds for the CPU/CUDA EPs). A user pipelining two GPU-backed stages on separate threads (e.g. two different models, or two documents processed by a worker pool) has no obvious reason to serialize and will hit this unpredictably.

GPUs used: NVIDIA RTX A2000 8GB Laptop GPU (driver 580.173.02), Intel Iris Xe Graphics (ADL GT2). Both enumerated as separate `OrtEpDevice`s via `OrtEnv.GetEpDevices()`, backend is Vulkan.

### To reproduce

Minimal, self-contained repro (no dependency on our application code) — two sessions load the same small ONNX model (any model should reproduce this; we used a tiny PP-OCRv6 text-detector export, DBNet backbone, ~1.7 MB, single float32 input `x: [-1,3,-1,-1]`, single output `[-1,1,-1,-1]` — happy to share the file if useful), each pinned to a device via `SessionOptions.AppendExecutionProvider` against an `OrtEpDevice` from the WebGPU EP, then `Run()` is called from two `Task.Run` threads concurrently via `Task.WaitAll`.

```csharp
using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.EP.WebGpu;
using Microsoft.ML.OnnxRuntime.Tensors;

string modelPath = args[0]; // any small model; single float input works
string mode = args.Length > 1 ? args[1] : "concurrent"; // "concurrent" | "sequential" | "same-device"

var env = OrtEnv.Instance();
env.RegisterExecutionProviderLibrary("webgpu_ep_registration", WebGpuEp.GetLibraryPath());
var devices = env.GetEpDevices().Where(d => d.EpName == WebGpuEp.GetEpName()).ToList();

InferenceSession Build(OrtEpDevice device)
{
using var opts = new SessionOptions();
opts.AppendExecutionProvider(env, new[] { device },
new Dictionary { ["ep.webgpuexecutionprovider.enableInt64"] = "1" });
return new InferenceSession(modelPath, opts);
}

var deviceA = devices[0];
var deviceB = mode == "same-device" ? devices[0] : devices[^1];

using var sessionA = Build(deviceA);
using var sessionB = Build(deviceB);

var inputMeta = sessionA.InputMetadata.First();
var tensor = new DenseTensor(new float[1 * 3 * 64 * 64], new[] { 1, 3, 64, 64 });
var inputs = new List { NamedOnnxValue.CreateFromTensor(inputMeta.Key, tensor) };

void RunOnce(InferenceSession s) { using var r = s.Run(inputs); _ = r.First().AsTensor().Length; }

if (mode == "sequential")
{
RunOnce(sessionA);
RunOnce(sessionB);
}
else
{
var tA = Task.Run(() => RunOnce(sessionA));
var tB = Task.Run(() => RunOnce(sessionB));
Task.WaitAll(tA, tB); // <-- segfaults here (mode="concurrent" or "same-device")
}
```

- `dotnet run -- model.onnx concurrent` → SIGSEGV, exit 139.
- `dotnet run -- model.onnx sequential` → completes cleanly, exit 0.
- `dotnet run -- model.onnx same-device` → also SIGSEGV, exit 139 (rules out "cross-GPU" as the specific trigger — plain cross-session concurrency is enough).

**Backtrace** (gdb `thread apply all bt` on the `concurrent` run). No debug symbols on the release binaries, but the crashing thread's frame 0 is a null-pointer call from within the WebGPU provider's `Run()` dispatch path:

```
Thread 44 ".NET TP Worker" received signal SIGSEGV, Segmentation fault.
#0 0x0000000000000000 in ?? ()
#1 0x00007fefb1c4cc60 in ?? () from .../libonnxruntime_providers_webgpu.so
#2 0x00007fefb1b03a11 in ?? () from .../libonnxruntime_providers_webgpu.so
#3 0x00007fefb1b05f68 in ?? () from .../libonnxruntime_providers_webgpu.so
#4 0x00007fefb352886b in ?? () from .../libonnxruntime.so
#5 0x00007fefb353656e in ?? () from .../libonnxruntime.so
#6 0x00007fefb34dce00 in ?? () from .../libonnxruntime.so
#7 0x00007fefb34dd49f in ?? () from .../libonnxruntime.so
#8 0x00007fefb34d7390 in ?? () from .../libonnxruntime.so
#9 0x00007fefb3503366 in ?? () from .../libonnxruntime.so
#10 0x00007fefb2daa22e in ?? () from .../libonnxruntime.so
#11 0x00007fefb2258ee6 in ?? () from .../libonnxruntime_providers_webgpu.so
#12 0x00007fefb1b8add4 in ?? () from .../libonnxruntime_providers_webgpu.so
#13 0x00007fefb1c5ba31 in ?? () from .../libonnxruntime_providers_webgpu.so
#14 0x00007fefb1b11bc0 in ?? () from .../libonnxruntime_providers_webgpu.so
#15 0x00007fefb2d6b3bd in ?? () from .../libonnxruntime.so
```

Another `.NET TP Worker` thread, at the same moment, is blocked inside the provider trying to lock a mutex through what looks like a garbage/corrupted pointer (`mutex=0x8`) — consistent with a data race on some shared WebGPU-EP state (an internal lock or handle being concurrently initialized/destroyed and accessed from the two threads):

```
Thread 49 ".NET TP Worker":
#0 ___pthread_mutex_lock (mutex=0x8) at ./nptl/pthread_mutex_lock.c:80
#1 0x00007fefb1e1a8ee in ?? () from .../libonnxruntime_providers_webgpu.so
#2 0x00007fefb1f0303f in ?? () from .../libonnxruntime_providers_webgpu.so
#3 0x00007fefb1ee336c in ?? () from .../libonnxruntime_providers_webgpu.so
#4 0x00007fefb1f1e42f in ?? () from .../libonnxruntime_providers_webgpu.so
#5 0x00007fefb1f1d812 in ?? () from .../libonnxruntime_providers_webgpu.so
```

Happy to rebuild with debug symbols, symbolicate this properly, or provide a full core dump if that would help narrow it down further.

### Urgency

Not blocking for us — we can serialize GPU inference on our side as a workaround. Flagging because it's a silent native crash with no documented thread-safety boundary, so other users pipelining multiple WebGPU-backed sessions across threads will likely hit it without warning.

### Platform

Linux

### OS Version

Ubuntu 24.04.5 LTS, kernel 7.0.0-31-generic

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

Microsoft.ML.OnnxRuntime 1.24.4, Microsoft.ML.OnnxRuntime.EP.WebGpu 0.3.0

### ONNX Runtime API

C#

### Architecture

X64

### Execution Provider

Other / Unknown (WebGPU plugin EP — not in the dropdown's list; Dawn backend, Vulkan)

### Execution Provider Library Version

Dawn (via `Microsoft.ML.OnnxRuntime.EP.WebGpu` 0.3.0), Vulkan backend

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.