microsoft / microsoft/onnxruntime

[Performance] QMoECPU<MLFloat16> intermittently livelocks (100% CPU, never returns) under multi-threaded intra-op execution on Apple Silicon

Open
#29,047 1 comment 1 reaction 2 assignees Claimed by @tianleiwu View on GitHub
performance
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

I'm running a token-classification model that uses the com.microsoft.QMoE contrib op (openai/privacy-filter, q4f16 ONNX export — it also uses GatherBlockQuantized and MatMulNBits). When I loop inference over a corpus of short documents, a random Run() call occasionally never returns: the process pins at 100% CPU inside a single QMoECPU::Compute call and stays there until killed (I've left one going for over 6 CPU-hours).

What I've observed:
- Repeated sample stack traces of the stuck process are identical every time: the main thread is inside QMoECPU::Compute(...)::'lambda0'(long)::operator() and the intra-op pool threads are in ThreadPoolTempl::WorkerLoop.
- Which document hangs is non-deterministic: one run hangs on document ~25, the next on ~67. Any individual "hanging" document completes in about 0.4 s when run alone in a fresh session, so it isn't an input pathology.
- With intra_op_num_threads = 1, the same 1000-document corpus completes start to finish every time — multiple full runs, zero hangs (~275 ms per document on an M-series CPU).

Multi-threaded-only + non-deterministic position + spinning at 100% with identical stacks looks like a synchronization race/livelock in the QMoE CPU kernel's parallel section rather than slow compute.

### To reproduce

To reproduce:

1. Model: https://huggingface.co/openai/privacy-filter → onnx/model_q4f16.onnx + onnx/model_q4f16.onnx_data (the fp16 export shows the same behavior for me, so it isn't specific to the 4-bit weights — every dtype of this model goes through QMoE).
2. Create a session with default options (the hang also reproduces with an explicit intra_op_num_threads = 6).
3. Loop Run() over ~1000 varied-length inputs (input_ids / attention_mask, int64, shape [1, n], n ≈ 30–600 — ordinary tokenized sentences).
4. Expected: each Run() returns in ~0.1–0.4 s. Actual: at a random iteration, one Run() never returns and the process spins at 100% CPU.
5. Workaround confirmed on my machine: set intra_op_num_threads = 1 — no hangs across repeated full-corpus runs, at ~3× the latency.

(I'm calling ORT through the Rust ort binding 2.0.0-rc.12, but the calls involved are plain C-API Run() semantics — nothing exotic.)

### Urgency

Blocks multi-threaded CPU inference for quantized MoE (gpt-oss-architecture) models on Apple Silicon. Single-thread workaround works but costs ~3× latency — moderately urgent for anyone deploying these models on macOS.

### Platform

Mac

### OS Version

macOS 26.x (Apple Silicon)

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.24 (via the ort Rust crate 2.0.0-rc.12). Note: also have Python onnxruntime 1.25.0 on the same machine; single inferences run fine there, but │ │ or Commit ID │ I haven't stress-tested multi-threaded 1.25 over the full corpus, so can't say if the race is fixed in 1.25.

### ONNX Runtime API

C

### Architecture

ARM64

### Execution Provider

Default CPU

### Execution Provider Library Version

(bundled with ORT 1.24 release)

### Model File

https://huggingface.co/openai/privacy-filter — onnx/model_q4f16.onnx + onnx/model_q4f16.onnx_data

### Is this a quantized model?

Yes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.