microsoft / microsoft/onnxruntime
WebGPU EP: concurrent Session.Run() across two sessions segfaults (Linux/Vulkan, same or different GPU)
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
Calling `Session.Run()` concurrently (from two separate threads, at the same time) on two different `InferenceSession` instances that both use the native **WebGPU plugin execution provider** (`Microsoft.ML.OnnxRuntime.EP.WebGpu`) segfaults the process (SIGSEGV / exit code 139), on **Linux via the Vulkan backend** (Dawn).
This reproduces:
- with the two sessions pinned to **two different physical GPUs** (an NVIDIA discrete GPU and an Intel integrated GPU, both enumerated via `OrtEnv.GetEpDevices()`), and
- with two sessions pinned to the **same** `OrtEpDevice`.
In both cases:
- Constructing the two sessions **concurrently** is fine (no crash).
- Running the two sessions' `Run()` calls **sequentially** (one completes before the other starts) is fine (no crash), even after concurrent construction.
- Only **concurrent `Run()`** calls crash.
This looks like the same underlying issue as #27592 / #31627 ("WebGPU EP crashes on macOS when running concurrent multi threaded inference," still open, last confirmed on ORT 1.28), but reproduced on a different Dawn backend (Vulkan/Linux instead of Metal/macOS) and additionally across two separate physical GPUs, which that issue doesn't appear to cover. Filing separately since the backend/platform and cross-GPU angle are new data points — happy to have this folded into #31627 instead if the maintainers prefer.
It's worth flagging as more than a minor edge case because **it's silent** — no exception, no ORT-level error, just a native crash — and nothing in the WebGPU EP's documentation states that concurrent `Run()` across sessions is unsafe (in contrast to the well-known guidance that a *single* session's `Run()` is safe to call concurrently, which holds for the CPU/CUDA EPs). A user pipelining two GPU-backed stages on separate threads (e.g. two different models, or two documents processed by a worker pool) has no obvious reason to serialize and will hit this unpredictably.
GPUs used: NVIDIA RTX A2000 8GB Laptop GPU (driver 580.173.02), Intel Iris Xe Graphics (ADL GT2). Both enumerated as separate `OrtEpDevice`s via `OrtEnv.GetEpDevices()`, backend is Vulkan.
### To reproduce
Minimal, self-contained repro (no dependency on our application code) — two sessions load the same small ONNX model (any model should reproduce this; we used a tiny PP-OCRv6 text-detector export, DBNet backbone, ~1.7 MB, single float32 input `x: [-1,3,-1,-1]`, single output `[-1,1,-1,-1]` — happy to share the file if useful), each pinned to a device via `SessionOptions.AppendExecutionProvider` against an `OrtEpDevice` from the WebGPU EP, then `Run()` is called from two `Task.Run` threads concurrently via `Task.WaitAll`.
```csharp
using Microsoft.ML.OnnxRuntime;
using Microsoft.ML.OnnxRuntime.EP.WebGpu;
using Microsoft.ML.OnnxRuntime.Tensors;
string modelPath = args[0]; // any small model; single float input works
string mode = args.Length > 1 ? args[1] : "concurrent"; // "concurrent" | "sequential" | "same-device"
var env = OrtEnv.Instance();
env.RegisterExecutionProviderLibrary("webgpu_ep_registration", WebGpuEp.GetLibraryPath());
var devices = env.GetEpDevices().Where(d => d.EpName == WebGpuEp.GetEpName()).ToList();
InferenceSession Build(OrtEpDevice device)
{
using var opts = new SessionOptions();
opts.AppendExecutionProvider(env, new[] { device },
new Dictionary { ["ep.webgpuexecutionprovider.enableInt64"] = "1" });
return new InferenceSession(modelPath, opts);
}
var deviceA = devices[0];
var deviceB = mode == "same-device" ? devices[0] : devices[^1];
using var sessionA = Build(deviceA);
using var sessionB = Build(deviceB);
var inputMeta = sessionA.InputMetadata.First();
var tensor = new DenseTensor(new float[1 * 3 * 64 * 64], new[] { 1, 3, 64, 64 });
var inputs = new List { NamedOnnxValue.CreateFromTensor(inputMeta.Key, tensor) };
void RunOnce(InferenceSession s) { using var r = s.Run(inputs); _ = r.First().AsTensor().Length; }
if (mode == "sequential")
{
RunOnce(sessionA);
RunOnce(sessionB);
}
else
{
var tA = Task.Run(() => RunOnce(sessionA));
var tB = Task.Run(() => RunOnce(sessionB));
Task.WaitAll(tA, tB); // <-- segfaults here (mode="concurrent" or "same-device")
}
```
- `dotnet run -- model.onnx concurrent` → SIGSEGV, exit 139.
- `dotnet run -- model.onnx sequential` → completes cleanly, exit 0.
- `dotnet run -- model.onnx same-device` → also SIGSEGV, exit 139 (rules out "cross-GPU" as the specific trigger — plain cross-session concurrency is enough).
**Backtrace** (gdb `thread apply all bt` on the `concurrent` run). No debug symbols on the release binaries, but the crashing thread's frame 0 is a null-pointer call from within the WebGPU provider's `Run()` dispatch path:
```
Thread 44 ".NET TP Worker" received signal SIGSEGV, Segmentation fault.
#0 0x0000000000000000 in ?? ()
#1 0x00007fefb1c4cc60 in ?? () from .../libonnxruntime_providers_webgpu.so
#2 0x00007fefb1b03a11 in ?? () from .../libonnxruntime_providers_webgpu.so
#3 0x00007fefb1b05f68 in ?? () from .../libonnxruntime_providers_webgpu.so
#4 0x00007fefb352886b in ?? () from .../libonnxruntime.so
#5 0x00007fefb353656e in ?? () from .../libonnxruntime.so
#6 0x00007fefb34dce00 in ?? () from .../libonnxruntime.so
#7 0x00007fefb34dd49f in ?? () from .../libonnxruntime.so
#8 0x00007fefb34d7390 in ?? () from .../libonnxruntime.so
#9 0x00007fefb3503366 in ?? () from .../libonnxruntime.so
#10 0x00007fefb2daa22e in ?? () from .../libonnxruntime.so
#11 0x00007fefb2258ee6 in ?? () from .../libonnxruntime_providers_webgpu.so
#12 0x00007fefb1b8add4 in ?? () from .../libonnxruntime_providers_webgpu.so
#13 0x00007fefb1c5ba31 in ?? () from .../libonnxruntime_providers_webgpu.so
#14 0x00007fefb1b11bc0 in ?? () from .../libonnxruntime_providers_webgpu.so
#15 0x00007fefb2d6b3bd in ?? () from .../libonnxruntime.so
```
Another `.NET TP Worker` thread, at the same moment, is blocked inside the provider trying to lock a mutex through what looks like a garbage/corrupted pointer (`mutex=0x8`) — consistent with a data race on some shared WebGPU-EP state (an internal lock or handle being concurrently initialized/destroyed and accessed from the two threads):
```
Thread 49 ".NET TP Worker":
#0 ___pthread_mutex_lock (mutex=0x8) at ./nptl/pthread_mutex_lock.c:80
#1 0x00007fefb1e1a8ee in ?? () from .../libonnxruntime_providers_webgpu.so
#2 0x00007fefb1f0303f in ?? () from .../libonnxruntime_providers_webgpu.so
#3 0x00007fefb1ee336c in ?? () from .../libonnxruntime_providers_webgpu.so
#4 0x00007fefb1f1e42f in ?? () from .../libonnxruntime_providers_webgpu.so
#5 0x00007fefb1f1d812 in ?? () from .../libonnxruntime_providers_webgpu.so
```
Happy to rebuild with debug symbols, symbolicate this properly, or provide a full core dump if that would help narrow it down further.
### Urgency
Not blocking for us — we can serialize GPU inference on our side as a workaround. Flagging because it's a silent native crash with no documented thread-safety boundary, so other users pipelining multiple WebGPU-backed sessions across threads will likely hit it without warning.
### Platform
Linux
### OS Version
Ubuntu 24.04.5 LTS, kernel 7.0.0-31-generic
### ONNX Runtime Installation
Released Package
### ONNX Runtime Version or Commit ID
Microsoft.ML.OnnxRuntime 1.24.4, Microsoft.ML.OnnxRuntime.EP.WebGpu 0.3.0
### ONNX Runtime API
C#
### Architecture
X64
### Execution Provider
Other / Unknown (WebGPU plugin EP — not in the dropdown's list; Dawn backend, Vulkan)
### Execution Provider Library Version
Dawn (via `Microsoft.ML.OnnxRuntime.EP.WebGpu` 0.3.0), Vulkan backend
Contributor guide
Assessment
This issue has not been assessed yet.