Process hard-crashes after exactly 2^14 = 16384 output NDArray allocations — floor(16384/num_outputs) inferences (all compute units; not fixed by releasing outputs or reloading the model)
- Dominant language
- Python
- Stars
- 152
- Forks
- 45
- Avg merge
- 1d 7m
- Merged PRs (30d)
- 12
Description
### Summary
Sustained inference exhausts IOSurface-backed `NDArray` output storage after roughly **8,000–9,000 calls** and the process dies with an **uncatchable** Swift precondition (`EXC_BREAKPOINT` / SIGTRAP). It reproduces on **every compute unit** (ANE, GPU, CPU-preferred), on two unrelated models, and **releasing the outputs and reloading the `AIModel` do not reclaim**.
This is a hard blocker for any long-running inference process: a service, a batch job, or a video pipeline will die after a few thousand frames regardless of what the caller does.
macOS 27.0 (26A5421a), Mac17,7 (M5 Max, 128 GB), `CoreAIRuntime 3600.83.2.14.1`, `ANEServices 10.19`, `coreai-core==1.0.0b2`, `coreai-torch==0.4.1`, Python 3.12.13.
### Reproduction
```python
import asyncio, numpy as np
from coreai.runtime import AIModel, NDArray, SpecializationOptions, ComputeUnitKind
async def main():
opts = SpecializationOptions.from_preferred_compute_unit_kind(ComputeUnitKind.neural_engine())
model = await AIModel.load("model.aimodel", specialization_options=opts)
fn = model.load_function(next(iter(model.function_names)))
x = NDArray(np.random.rand(1, 3, 224, 224).astype(np.float16))
for i in range(1, 20001):
await fn({"pixel_values": x}) # result discarded immediately
if i % 1000 == 0:
print(i, flush=True)
asyncio.run(main())
```
Dies between 8,000 and 9,000 every time.
### Observed
Two assertion sites, depending on the model's output shape:
```
CoreAIRuntime/NDArray+SharedStorage.swift:108: Fatal error: Failed to allocate storage for
NDArray with requirements: NDArrayDescriptor(scalarType: float16, shape: [1, 20, 56, 56],
alignments: [1, 1, 1, 32, 1], interleave: [1, 1, 1, 1], ordering: [0, 1, 2, 3],
storageKind: ioSurface)
```
```
CoreAIRuntime/NDArray+Pool.swift:77: Fatal error: Failed to allocate storage for NDArray
with byteCount: 1572864, sk: ioSurface, st: float16
```
Crash thread:
```
0 libswiftCore.dylib _assertionFailure(_:_:file:line:flags:) + 216
1 CoreAIRuntime 0x… + 494168
…
12 libswift_Concurrency.dylib completeTaskWithClosure(...)
Exception Type: EXC_BREAKPOINT (SIGTRAP)
```
### What was ruled out
| Hypothesis | Result |
|---|---|
| ANE-specific | **No.** Reproduces on the GPU lane at the same count. |
| Caller retaining results | **No.** Results are discarded (`del` / never bound); `gc.collect()` every 100 iterations changes nothing. |
| Per-`AIModel` accumulation | **No.** Reloading the model and re-fetching the function every 2,000 calls still dies at the same point. |
| Memory pressure | **No.** VM report at death shows **Writable regions Total ≈ 308 MB** on a 128 GB machine. |
| Model-specific | **No.** Two unrelated models (a 1.2 M-param SR convnet with one `[1,3,512,512]` output; a small ViT segmentation model with two outputs) both die. |
| System-wide accumulation | **No.** Each fresh process gets its own ~8,000, so it is per-process and resets on exit. |
That combination points at IOSurface **handles** rather than bytes — a per-process IOSurface limit being reached because output-backing surfaces are not returned to the pool.
### Expected
Either the output storage is recycled so a steady-state inference loop runs indefinitely, or — at minimum — the failure surfaces as a **catchable error** rather than a Swift precondition. As it stands the process cannot defend itself: there is nothing to catch, and no API to drain or bound the pool.
### Impact
Any process doing sustained inference will terminate after a few thousand calls. At the ~1 ms/inference this model achieves, that is under 10 seconds of continuous work. The only workaround we have found is to shard the workload across subprocesses and restart before ~8,000 calls, which is not viable for a latency-sensitive or stateful service.
### Possibly related
`#11` (runtime clobbers an unrelated live tensor) also involves runtime storage lifetime, though the symptom there is corruption rather than exhaustion.
Contributor guide
Research direction
Run the provided Python reproduction with the listed compute units and confirm the repeatable allocation threshold. Then investigate the reported CoreAIRuntime/NDArray+SharedStorage.swift:108 and NDArray+Pool.swift:77 assertion sites, along with related storage-lifetime behavior in issue #11. Done means sustained inference no longer exhausts output storage, or the failure is exposed as a catchable error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100