microsoft / microsoft/onnxruntime
[Performance] QMoECPU<MLFloat16> intermittently livelocks (100% CPU, never returns) under multi-threaded intra-op execution on Apple Silicon
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
I'm running a token-classification model that uses the com.microsoft.QMoE contrib op (openai/privacy-filter, q4f16 ONNX export — it also uses GatherBlockQuantized and MatMulNBits). When I loop inference over a corpus of short documents, a random Run() call occasionally never returns: the process pins at 100% CPU inside a single QMoECPU::Compute call and stays there until killed (I've left one going for over 6 CPU-hours).
What I've observed:
- Repeated sample stack traces of the stuck process are identical every time: the main thread is inside QMoECPU::Compute(...)::'lambda0'(long)::operator() and the intra-op pool threads are in ThreadPoolTempl::WorkerLoop.
- Which document hangs is non-deterministic: one run hangs on document ~25, the next on ~67. Any individual "hanging" document completes in about 0.4 s when run alone in a fresh session, so it isn't an input pathology.
- With intra_op_num_threads = 1, the same 1000-document corpus completes start to finish every time — multiple full runs, zero hangs (~275 ms per document on an M-series CPU).
Multi-threaded-only + non-deterministic position + spinning at 100% with identical stacks looks like a synchronization race/livelock in the QMoE CPU kernel's parallel section rather than slow compute.
### To reproduce
To reproduce:
1. Model: https://huggingface.co/openai/privacy-filter → onnx/model_q4f16.onnx + onnx/model_q4f16.onnx_data (the fp16 export shows the same behavior for me, so it isn't specific to the 4-bit weights — every dtype of this model goes through QMoE).
2. Create a session with default options (the hang also reproduces with an explicit intra_op_num_threads = 6).
3. Loop Run() over ~1000 varied-length inputs (input_ids / attention_mask, int64, shape [1, n], n ≈ 30–600 — ordinary tokenized sentences).
4. Expected: each Run() returns in ~0.1–0.4 s. Actual: at a random iteration, one Run() never returns and the process spins at 100% CPU.
5. Workaround confirmed on my machine: set intra_op_num_threads = 1 — no hangs across repeated full-corpus runs, at ~3× the latency.
(I'm calling ORT through the Rust ort binding 2.0.0-rc.12, but the calls involved are plain C-API Run() semantics — nothing exotic.)
### Urgency
Blocks multi-threaded CPU inference for quantized MoE (gpt-oss-architecture) models on Apple Silicon. Single-thread workaround works but costs ~3× latency — moderately urgent for anyone deploying these models on macOS.
### Platform
Mac
### OS Version
macOS 26.x (Apple Silicon)
### ONNX Runtime Installation
Released Package
### ONNX Runtime Version or Commit ID
1.24 (via the ort Rust crate 2.0.0-rc.12). Note: also have Python onnxruntime 1.25.0 on the same machine; single inferences run fine there, but │ │ or Commit ID │ I haven't stress-tested multi-threaded 1.25 over the full corpus, so can't say if the race is fixed in 1.25.
### ONNX Runtime API
C
### Architecture
ARM64
### Execution Provider
Default CPU
### Execution Provider Library Version
(bundled with ORT 1.24 release)
### Model File
https://huggingface.co/openai/privacy-filter — onnx/model_q4f16.onnx + onnx/model_q4f16.onnx_data
### Is this a quantized model?
Yes
Contributor guide
Assessment
This issue has not been assessed yet.