microsoft / microsoft/onnxruntime
WebGPU EP crashes on macOS when running concurrent multi threaded inference
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
**Note:** Re opening this issue as a follow up to #27592, which was automatically closed by the stale bot despite being an active (and I can't click to re open), reproducible bug. Confirmed still present on latest ORT (`v1.28`) so wanted to create a new anyway.
The WebGPU Execution Provider crashes on macOS (Apple Silicon) when multiple threads create sessions and run inference concurrently. The crash is non-deterministic and manifests as different errors on each run, indicating a race condition in Dawn's Metal backend.
Observed crashes (non-deterministic, varies per run)
Run 1 — Metal assertion:
```
**-[AGXG15GFamilyCommandBuffer blitCommandEncoderCommon:]:832:**
failed assertion `A command encoder is already encoding to this command buffer'
```
**Run 2 — Memory corruption:**
` malloc: *** error for object 0x784797e90: pointer being freed was not allocated
`
**Run 3 — Trace trap:**
` [1] 56040 trace trap cargo run --release -- version-RFB-640.onnx`
Notes:
- Single-threaded session creation and inference works fine - Sequential (non-concurrent) multi-session usage works fine
- The same workload works correctly with the CoreML EP
### To reproduce
minimal Repo: https://github.com/altunenes/ort-webgpu-thread-crash
git clone https://github.com/altunenes/ort-webgpu-thread-crash.git
cargo run --release -- version-RFB-640.onnx
(simple face detection model)
The repro creates 2 persistent WebGPU sessions, then spawns 5 threads that each create a session and run inference concurrently while the main thread also runs inference on the persistent sessions.
### Urgency
_No response_
### Platform
Mac
### OS Version
26.2
### ONNX Runtime Installation
Built from Source
### ONNX Runtime Version or Commit ID
1.28
### ONNX Runtime API
Other / Unknown
### Architecture
X64
### Execution Provider
Other / Unknown
### Execution Provider Library Version
_No response_
Contributor guide
Research direction
Start with the linked minimal repository and run `cargo run --release version-RFB-640.onnx` on macOS to reproduce the concurrent WebGPU session crash. Compare the concurrent workload with single-threaded and sequential multi-session runs, then verify that concurrent inference completes without the reported Metal assertions, memory errors, or trace traps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, macos, rust
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100