apple / apple/coreai-torch

Error occurred when loading ANE module (MPSGraphExecutable.mm:3543; appleneuralengine Code=53 "Program load failed — no memory") — FP16 asset aborts the process on ANE load instead of returning an error

Open
#67 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
152
Forks
45
Avg merge
1d 7m
Merged PRs (30d)
12

Description

## What happens

Loading some FP16 `.aimodel` assets fails to load their ANE program, and MPSGraph then raises a failed assertion rather than returning an error or falling back. The process aborts. Nothing the caller writes can catch it.

```
Error = Error Domain=com.apple.appleneuralengine Code=53
"createProgramInstanceForModel:...: Program load failed — no memory (transient; retry under lower
memory pressure) (underlying=0x1)" UserInfo={_ANEErrorUnderlyingStatus=1, _ANEErrorLoadStage=4}

MPSGraphExecutable.mm:3543: failed assertion `Error occurred when loading ANE module:
Error Domain=MPSGraph Code=-1 "MPSGraphExecutable_Project.h:510:: could not load module from
MPSGraphPackage"'
```

The same asset loads and runs with `SpecializationOptions.cpu_only()`. So this is the ANE program specifically, and the abort is instead of a fallback.

The message says the condition is transient and suggests retrying. It is not transient here: it reproduces on every run, in a fresh process, on an idle machine with memory free. Two different graphs reproduce it, and one of them is 12 MB.

## Environment

M4 Max, macOS 27.0 beta 6 (26A5416b), `coreai-torch` 0.4.2, `coreai-core` 1.0.0b2, Python 3.12.

## Reproducer

```python
import torch, coreai_torch, asyncio, inspect, numpy as np
from pathlib import Path
from coreai_torch import TorchConverter
from coreai.runtime import AIModel, NDArray, SpecializationOptions

class Net(torch.nn.Module):
def __init__(self):
super().__init__()
ch = [3, 32, 64, 128, 128]
self.blocks = torch.nn.Sequential(*[
torch.nn.Sequential(torch.nn.Conv2d(ch[i], ch[i + 1], 3, stride=2, padding=1),
torch.nn.SiLU())
for i in range(4)
])
self.head = torch.nn.Conv2d(128, 80, 1)

def forward(self, x):
s = self.head(self.blocks(x)).flatten(2).transpose(1, 2).max(dim=-1)[0]
return s.topk(300, dim=1)[0]

net = Net().eval().half()
ex = torch.rand(1, 3, 640, 640).half()
with torch.no_grad():
ep = torch.export.export(net, (ex,)).run_decompositions(coreai_torch.get_decomp_table())
c = TorchConverter()
c.add_exported_program(ep, entrypoint_name="main", input_names=["image"], output_names=["out0"])
prog = c.to_coreai(); prog.optimize(); prog.save_asset(Path("fp16.aimodel"))

async def go(options=None):
m = AIModel.load(Path("fp16.aimodel"), options) if options else AIModel.load(Path("fp16.aimodel"))
m = await m if inspect.isawaitable(m) else m
f = m.load_function("main")
f = await f if inspect.isawaitable(f) else f
o = f({"image": NDArray(np.random.rand(1, 3, 640, 640).astype(np.float16))})
return await o if inspect.isawaitable(o) else o

asyncio.run(go(SpecializationOptions.cpu_only())) # OK
asyncio.run(go()) # aborts
```

I hit this on two real graphs first — an FP16 YOLO26-pose export at 640 and a smaller detection export at 32 — before reducing it to the above. The FP32 versions of both load and run on the ANE without complaint, and several other FP16 exports of similar size do too, so it is not every FP16 asset.

## What I am asking for

An error instead of an abort. A caller can retry, fall back to the GPU, or report the problem, but only if `AIModel.load` returns rather than aborting the process. As it stands, an application that ships an FP16 asset can be terminated by a runtime condition it cannot detect or handle.

Whether the ANE program should load at all is the second question, and the more useful one if the "no memory" diagnosis is wrong — the machine has memory, and the condition does not clear on retry.

Context: found while adding a `format="coreai"` export to Ultralytics ([ultralytics/ultralytics#25926](https://github.com/ultralytics/ultralytics/pull/25926)). That PR exports FP16 but does not load it back, for this reason.

Contributor guide

Open the contributing guide

Research direction

Start at the AIModel.load path shown in the reproducer and compare default ANE loading with SpecializationOptions.cpu_only(). Use the provided FP16 model-generation script to reproduce the failure; done means the ANE load returns an error or supports fallback instead of aborting the process.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.