apple / apple/coreai-torch

debugging comparator cannot execute the source ExportedProgram — TypeError: 'int' object is not callable in _guards_fn (0.4.1 and 0.4.2)

Open
#76 0 comments 0 reactions 1 assignee Claimed by @cymbalrush View on GitHub
Dominant language
Python
Stars
152
Forks
45
Avg merge
1d 7m
Merged PRs (30d)
12

Description

### Summary

`coreai_torch.debugging.comparator.create_comparator_for_programs` cannot execute the source `ExportedProgram`, so the op-by-op comparator is unusable. It fails with `TypeError: 'int' object is not callable` while running the exported module's `_guards_fn`.

Reproduces on **both** shipping combinations: `coreai-torch 0.4.1` / torch 2.11.0 and `coreai-torch 0.4.2` / torch 2.13.0. macOS 27.0 (26A5421a), `CoreAIRuntime 3600.83.2.14.1`, `coreai-core 1.0.0b2`.

This matters more than a typical tooling bug because of what it blocks — see **Why this is load-bearing** below.

### Reproduction

```python
import asyncio, torch, torch.nn as nn
from coreai_torch import TorchConverter, get_decomp_table
from coreai_torch.debugging.comparator import create_comparator_for_programs, Comparator

class M(nn.Module):
def __init__(self):
super().__init__(); self.c = nn.Conv2d(3, 8, 3, 1, 1); self.r = nn.ReLU()
def forward(self, x):
return self.r(self.c(x))

async def main():
torch.manual_seed(0)
m = M().eval(); x = torch.rand(1, 3, 16, 16)
ep = torch.export.export(m, args=(x,)).run_decompositions(dict(get_decomp_table()))
prog = TorchConverter().add_exported_program(
ep, input_names=["x"], output_names=["out"]).to_coreai()
cmp = await create_comparator_for_programs(ep, prog, "main")
await cmp.compare(lambda *_: Comparator.Status.PASS, {"x": x.numpy()})

asyncio.run(main())
```

### Observed

```
TypeError: 'int' object is not callable

While executing %_guards_fn : [num_users=0] = call_module[target=_guards_fn](args = (%x,), kwargs = {})
Original traceback:
None
```

The failure is in executing the **source** program (`torch.fx` side), not in anything Core AI computes.

### Why this is load-bearing

The comparator is not just a convenience here — for **ANE numerics questions it is the only faithful instrument**, and I would like to explain why, since it also argues for prioritising this.

The obvious alternative is to instrument the model: re-export with intermediate tensors added as extra outputs, then compare layer by layer. **That does not work on this stack.** Adding outputs changes fusion, and in my case it changed ANE *eligibility* outright — a graph that compiled and ran ANE-resident stopped compiling once probe outputs were added:

```
_ANECompiler : ANECCompile() FAILED
Compiler internal error: It has to be valid custom strides
- From PEFUSED_GOC Layer: … TERNARY_DYNAMIC_GOC: Pre-Scale: 1, ScaleBiasNegate: Y,
ScaleBiasBroadcast: [ W:192 ]
```

The ANE lane then silently fell back, and the instrumented run reported ANE numbers identical to the GPU numbers to the last decimal. Truncation and prefix models have the same problem: any re-export is a different artifact.

So to investigate a discrepancy that appears **only on one compute unit**, one needs to read intermediates out of the *same compiled asset* — which is exactly what `CoreAIInspector` / `create_comparator_for_programs` are for, and `create_comparator_for_programs` even accepts `specialization_options` so it can target a lane. With it broken, there is no supported way to localize a lane-specific numerical difference.

Concretely, this is blocking me on a real case: a small ViT segmentation model measures **fp32 134 dB**, **fp16 GPU 66.62 dB**, and **fp16 ANE 43.58 dB** against the same reference, from the *identical* `.aimodel`. Same bytes, same weights, 23 dB between two lanes. I cannot localize it.

### Minor issues noticed alongside

Not bugs as such, but they cost time and may be worth a docs pass:

- Nothing is exported at the `coreai_torch.debugging` package level — `create_comparator_for_programs` must be imported from `.comparator`, and the docs' `create_validator_for_*` names are likewise only in `.validator`.
- `create_comparator_for_programs` and `Comparator.compare` are both **coroutines**; the documentation examples do not show `await`.
- `Comparator.Status` has only `PASS` / `FAIL` / `UNKNOWN`. A `SKIP` would be useful for nodes whose outputs are not comparable (differing shapes, non-tensor results) — currently they have to be reported as `PASS` or `UNKNOWN`, neither of which is accurate.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.