apple / apple/coreai-torch

Process hard-crashes after exactly 2^14 = 16384 output NDArray allocations — floor(16384/num_outputs) inferences (all compute units; not fixed by releasing outputs or reloading the model)

Open
#75 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
152
Forks
45
Avg merge
1d 7m
Merged PRs (30d)
12

Description

### Summary

Sustained inference exhausts IOSurface-backed `NDArray` output storage after roughly **8,000–9,000 calls** and the process dies with an **uncatchable** Swift precondition (`EXC_BREAKPOINT` / SIGTRAP). It reproduces on **every compute unit** (ANE, GPU, CPU-preferred), on two unrelated models, and **releasing the outputs and reloading the `AIModel` do not reclaim**.

This is a hard blocker for any long-running inference process: a service, a batch job, or a video pipeline will die after a few thousand frames regardless of what the caller does.

macOS 27.0 (26A5421a), Mac17,7 (M5 Max, 128 GB), `CoreAIRuntime 3600.83.2.14.1`, `ANEServices 10.19`, `coreai-core==1.0.0b2`, `coreai-torch==0.4.1`, Python 3.12.13.

### Reproduction

```python
import asyncio, numpy as np
from coreai.runtime import AIModel, NDArray, SpecializationOptions, ComputeUnitKind

async def main():
opts = SpecializationOptions.from_preferred_compute_unit_kind(ComputeUnitKind.neural_engine())
model = await AIModel.load("model.aimodel", specialization_options=opts)
fn = model.load_function(next(iter(model.function_names)))
x = NDArray(np.random.rand(1, 3, 224, 224).astype(np.float16))
for i in range(1, 20001):
await fn({"pixel_values": x}) # result discarded immediately
if i % 1000 == 0:
print(i, flush=True)

asyncio.run(main())
```

Dies between 8,000 and 9,000 every time.

### Observed

Two assertion sites, depending on the model's output shape:

```
CoreAIRuntime/NDArray+SharedStorage.swift:108: Fatal error: Failed to allocate storage for
NDArray with requirements: NDArrayDescriptor(scalarType: float16, shape: [1, 20, 56, 56],
alignments: [1, 1, 1, 32, 1], interleave: [1, 1, 1, 1], ordering: [0, 1, 2, 3],
storageKind: ioSurface)
```

```
CoreAIRuntime/NDArray+Pool.swift:77: Fatal error: Failed to allocate storage for NDArray
with byteCount: 1572864, sk: ioSurface, st: float16
```

Crash thread:

```
0 libswiftCore.dylib _assertionFailure(_:_:file:line:flags:) + 216
1 CoreAIRuntime 0x… + 494168

12 libswift_Concurrency.dylib completeTaskWithClosure(...)
Exception Type: EXC_BREAKPOINT (SIGTRAP)
```

### What was ruled out

| Hypothesis | Result |
|---|---|
| ANE-specific | **No.** Reproduces on the GPU lane at the same count. |
| Caller retaining results | **No.** Results are discarded (`del` / never bound); `gc.collect()` every 100 iterations changes nothing. |
| Per-`AIModel` accumulation | **No.** Reloading the model and re-fetching the function every 2,000 calls still dies at the same point. |
| Memory pressure | **No.** VM report at death shows **Writable regions Total ≈ 308 MB** on a 128 GB machine. |
| Model-specific | **No.** Two unrelated models (a 1.2 M-param SR convnet with one `[1,3,512,512]` output; a small ViT segmentation model with two outputs) both die. |
| System-wide accumulation | **No.** Each fresh process gets its own ~8,000, so it is per-process and resets on exit. |

That combination points at IOSurface **handles** rather than bytes — a per-process IOSurface limit being reached because output-backing surfaces are not returned to the pool.

### Expected

Either the output storage is recycled so a steady-state inference loop runs indefinitely, or — at minimum — the failure surfaces as a **catchable error** rather than a Swift precondition. As it stands the process cannot defend itself: there is nothing to catch, and no API to drain or bound the pool.

### Impact

Any process doing sustained inference will terminate after a few thousand calls. At the ~1 ms/inference this model achieves, that is under 10 seconds of continuous work. The only workaround we have found is to shard the workload across subprocesses and restart before ~8,000 calls, which is not viable for a latency-sensitive or stateful service.

### Possibly related

`#11` (runtime clobbers an unrelated live tensor) also involves runtime storage lifetime, though the symptom there is corruption rather than exhaustion.

Contributor guide

Open the contributing guide

Research direction

Run the provided Python reproduction with the listed compute units and confirm the repeatable allocation threshold. Then investigate the reported CoreAIRuntime/NDArray+SharedStorage.swift:108 and NDArray+Pool.swift:77 assertion sites, along with related storage-lifetime behavior in issue #11. Done means sustained inference no longer exhausts output storage, or the failure is exposed as a catchable error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.