microsoft / microsoft/onnxruntime

XNNPACK EP: concurrent InferenceSession creation races the lazily-created stored allocator (use-after-free)

Open
#32,461 4 comments 0 reactions 0 assignees View on GitHub
ep:Xnnpack platform:mobile
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

The XNNPACK execution provider keeps a single process-wide "stored allocator" that it creates lazily behind an **unsynchronised** null check. When two `InferenceSession`s are created concurrently (each getting its own `XnnpackExecutionProvider`) before any XNNPACK session has been created in the process, the two `CreatePreferredAllocators()` calls race, one allocator is freed while XNNPACK is left holding a pointer into it, and every subsequent XNNPACK allocation/free in the process is a use-after-free.

The relevant code:

`onnxruntime/core/providers/xnnpack/xnnpack_execution_provider.cc`:

```cpp
std::vector XnnpackExecutionProvider::CreatePreferredAllocators() {
const auto& [stored_allocator, xnn_allocator] = GetStoredAllocator();
if (!stored_allocator) { // <-- unsynchronised read
const AllocatorCreationInfo allocator_info(...);
stored_allocator = CreateAllocator(allocator_info); // <-- unsynchronised write to a shared static
}
xnn_allocator->context = stored_allocator.get(); // <-- shared static mutated by both threads
const xnn_status st = xnn_initialize(xnn_allocator); // XNNPACK copies the pointer once (XNN_INIT_ONCE)
...
}
```

`onnxruntime/core/providers/xnnpack/xnnpack_init.cc`:

```cpp
std::pair GetStoredAllocator() {
static AllocatorPtr ort_allocator; // shared across all EP instances
static xnn_allocator xnn_allocator_wrapper_ = { ort_allocator.get(), ... };
return {ort_allocator, &xnn_allocator_wrapper_};
}
```

Two threads can both observe `!stored_allocator`, both `CreateAllocator`, and the second assignment releases the first `shared_ptr`. Because `xnn_initialize` records the allocator only on the first call (`XNN_INIT_ONCE` / a CAS on `init_allocator`), XNNPACK can end up permanently pointing at the `IAllocator` owned by the freed `shared_ptr`. From then on, `xnn_allocate` / `xnn_aligned_allocate` / `xnn_deallocate` dereference a freed `IAllocator*`.

Observed crash signature (Apple, symbolicated): `EXC_BAD_ACCESS` in `onnxruntime::xnnpack::(anonymous namespace)::xnn_aligned_allocate` at the virtual `IAllocator::Alloc` call through the context pointer. Because the corrupted pointer lives for the life of the process, the fault surfaces in whichever XNNPACK op runs next, most visibly:
- session creation, in `Gemm::PrePack` / `MatMul::PrePack` -> `xnn_create_fully_connected_nc_f32` -> `xnn_aligned_allocate`
- inference, in `reshape_fully_connected_nc`
- session release, in `xnn_destroy_operator` -> `xnn_deallocate`

### To reproduce

On a fresh process, create several sessions on separate threads at once, each appending the XNNPACK EP, e.g.:

```cpp
Ort::Env env(ORT_LOGGING_LEVEL_ERROR, "race");
std::vector pool;
std::vector> sessions(N);
for (int t = 0; t < N; ++t) {
pool.emplace_back([&, t]{
Ort::SessionOptions so;
so.AppendExecutionProvider("XNNPACK", {{"intra_op_num_threads", "1"}});
// line all threads up so construction starts simultaneously, then:
sessions[t] = std::make_shared(env, model_path, so);
});
}
for (auto& th : pool) th.join();
```

Reproduced deterministically on macOS arm64 (ORT 1.27.0) with `MallocScribble=1` so the freed allocator faults immediately: with N=8 it crashes ~40-60% of runs; serialising session construction gives 0 crashes across hundreds of runs. Without heap poisoning the use-after-free is latent (reads stale-but-valid memory) and surfaces intermittently, which matches how it appears in the field. The same code is present in 1.29.0 and on `main`.

### Urgency

Moderate. It is intermittent and only affects concurrent first-time session creation, but when it triggers it corrupts a process-global pointer and crashes unpredictably for the rest of the process's life.

### Platform

Mac (reproduced), also affects iOS and Android (shared code).

### OS Version

macOS 15 (arm64); observed in production on iOS.

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.27.0 (also 1.29.0 and current main)

### ONNX Runtime API

C++

### Architecture

ARM64

### Execution Provider

Other / possibly contributes (XNNPACK)

### Suggested fix

Make the lazy creation of the stored allocator (and the `xnn_allocator->context` write + `xnn_initialize`) thread-safe, e.g. a `std::call_once` / mutex inside `GetStoredAllocator()` or around the init block in `CreatePreferredAllocators()`. As a consumer-side workaround, serialising `Ort::Session` construction with a process-wide mutex avoids it.

Contributor guide

Open the contributing guide

Research direction

Start with onnxruntime/core/providers/xnnpack/xnnpack_execution_provider.cc and xnnpack_init.cc, focusing on GetStoredAllocator() and CreatePreferredAllocators(). Run the concurrent session-creation reproduction on a fresh process, using the reported macOS setup if available. Done means first-time concurrent XNNPACK session creation no longer leaves XNNPACK holding a freed allocator, with the existing reproduction no longer crashing.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.