microsoft / microsoft/onnxruntime
[CoreML] Wrong FP16 output with RequireStaticInputShapes=1 on macOS 15 ARM64
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
## Summary
ONNX Runtime 1.30.0 returns stable but incorrect FP16 semantic scores from the CoreML execution provider on macOS 15 ARM64. The public reproducer uses:
```text
ModelFormat=MLProgram
RequireStaticInputShapes=1
```
The inference call returns success and finite values:
```text
Expected: [6.3321176 4.6452603 4.445041]
Actual: [5.518619 4.61189 4.4114037]
```
The same model and inputs pass the existing `0.01` limit with the ONNX Runtime CPU EP and the legacy CoreML `NeuralNetwork` format on the same host. The same static `MLProgram` path also passes on macOS 26 ARM64.
For this graph, changing only `RequireStaticInputShapes` to `0` also passes. This isolates a partition trigger, not the root cause.
## Reproduce
The public model, fixed inputs, and tests are in [`dualeai/seek` commit `8c89a567`](https://github.com/dualeai/seek/tree/8c89a56732033ecef5ed5caad5b62a9faea086c4).
Run this on macOS 15 ARM64:
```sh
git clone https://github.com/dualeai/seek.git
cd seek
git checkout 8c89a56732033ecef5ed5caad5b62a9faea086c4
make tokenizer-native
go test ./cmd/seek -run '^TestLateOnInferenceMatchesReferenceScores$' -count=1 -v
```
The test fails with the expected and actual vectors above.
Run the full provider matrix with:
```sh
go test ./cmd/seek -run '^TestLateOnCoreMLProviderMatrix$' -count=1 -v
```
The matrix test reports each completed case as a test error so that CI keeps all values. The [complete macOS 15 ARM64 log](https://github.com/dualeai/seek/actions/runs/34705027259/job/103583843652) contains the results below.
## Results
All rows use the same model and inputs on macOS 15.7.9 ARM64.
| Configuration | Scores | Result |
|---|---|---|
| `MLProgram`, static-only, `ALL`, full optimization | `[5.518619 4.61189 4.4114037]` | Wrong |
| `MLProgram`, static-only, `ALL`, basic optimization | `[5.518619 4.61189 4.4114037]` | Wrong |
| `MLProgram`, static-only, `CPUOnly`, full optimization | `[5.518619 4.61189 4.4114037]` | Wrong |
| `MLProgram`, non-static admission, `ALL`, full optimization | `[6.3361335 4.6472425 4.4510884]` | Passes |
| `NeuralNetwork`, static-only, `ALL`, full optimization | `[6.330186 4.644894 4.4470415]` | Passes |
| ONNX Runtime CPU EP, full optimization | `[6.330186 4.644894 4.4470415]` | Passes |
These controls show:
- `CPUOnly` gives the same wrong result, so the fault is not limited to GPU or Neural Engine execution.
- Basic and full optimization give the same wrong result, so full optimization is not required.
- Padded and full batches agree exactly.
- The no-optimization case stops at a `MatMul` shape error, so it is not a value control.
## Known fault boundary
The model has fixed external inputs, but ONNX shape inference leaves some intermediate dimensions unknown. For the original graph:
- Static-only admission assigns 467 of 474 optimized nodes to five CoreML partitions.
- Non-static admission assigns 468 nodes to four CoreML partitions.
- The extra CoreML node is `/bert/Where_2`. Its assignment merges two partitions and removes two inserted precision casts.
Later affected-host A/B tests changed this mask path and still returned the same wrong vector. These tests rule out `/bert/Where_2` and the tested mask additions as the only faulty operation. They do not identify the internal fault.
I have not reduced the fault to one operator. The internal cause can be in ONNX Runtime partitioning or lowering, or in the macOS 15 Core ML compiler. For callers, the CoreML EP claims a partition and returns changed output without an error.
Trace, source, and additional A/B evidence
The opset-14 input model contains decomposed FP16 normalization and no `LayerNormalization` node. A CPU optimizer dump keeps it decomposed with basic optimization. Full optimization [creates `SimplifiedLayerNormalization`](https://github.com/microsoft/onnxruntime/blob/f2c39fe2f838cf35ce7da92824f5a5e3ee6e88a7/onnxruntime/core/optimizer/layer_norm_fusion.cc#L819-L840) for the `Pow`-through-`Mul` parts. The CoreML [builder factory does not register that operator](https://github.com/microsoft/onnxruntime/blob/f2c39fe2f838cf35ce7da92824f5a5e3ee6e88a7/onnxruntime/core/providers/coreml/builders/op_builder_factory.cc#L66-L70). Both optimization levels fail, so this fusion is not required.
The [layer trace](https://github.com/dualeai/seek/actions/runs/34713083291/job/103605522163) first measures a difference at the embedding normalization output. Mean absolute error grows from `0.0023`-`0.0084` after layer 0 to `1.714`-`2.443` after layer 6. One active projection component changes from about `-0.386` to zero. Layer-0 residual values reach about `2,180`, so FP16 variance overflow is a possible clue. The trace does not expose the centered input or `Pow` output, so it does not prove this cause. The trace source is in [`dualeai/seek` commit `79970bea`](https://github.com/dualeai/seek/tree/79970bea7b3cb5eebcabc5353a618d5eb4cea511).
`RequireStaticInputShapes` feeds the CoreML [per-node support check](https://github.com/microsoft/onnxruntime/blob/f2c39fe2f838cf35ce7da92824f5a5e3ee6e88a7/onnxruntime/core/providers/coreml/coreml_execution_provider.cc#L55-L58), which [rejects a node with a dynamic inferred input shape](https://github.com/microsoft/onnxruntime/blob/f2c39fe2f838cf35ce7da92824f5a5e3ee6e88a7/onnxruntime/core/providers/coreml/builders/helper.cc#L47-L75). This explains the partition change, but not the wrong calculation.
Additional macOS 15 ARM64 A/B logs:
- [Static mask](https://github.com/dualeai/seek/actions/runs/34706705939/job/103588171135)
- [Static option disabled for the fixed-mask graph](https://github.com/dualeai/seek/actions/runs/34707299241/job/103589834168)
- [Run-time mask with fixed shape metadata](https://github.com/dualeai/seek/actions/runs/34708899368/job/103594186900)
- [Local mask additions on the CPU EP](https://github.com/dualeai/seek/actions/runs/34709877760/job/103596809433)
- [Padding-mask additions on the CPU EP](https://github.com/dualeai/seek/actions/runs/34710672421/job/103599017619)
## Expected behavior
CoreML must match the CPU EP within the normal FP16 tolerance. If CoreML cannot preserve ONNX semantics, the CoreML EP must not claim the partition.
## Impact and current project handling
The failure changes semantic search ranking without an error. Seek now skips only [this score-parity test](https://github.com/dualeai/seek/commit/7a89b0d55605ba60d4b78d277a43368156e5183d) on macOS 15 ARM64. All other tests remain active. This test exception does not mark the wrong vector as correct, and no model or runtime workaround remains in the project.
## System information
- OS: macOS 15.7.9, build 24G830, Darwin 24.6.0
- Runner image: GitHub `macos-15-arm64` 20260907.0337.1
- Hardware: Apple M1 ARM64 runner, 3 CPUs, 7 GiB RAM
- ONNX Runtime: 1.30.0, official `onnxruntime-osx-arm64` package
- API: C API through `onnxruntime_go` 1.36.0
- Go: 1.27.0, `darwin/arm64`
- CoreML options: `ModelFormat=MLProgram`, `RequireStaticInputShapes=1`, `MLComputeUnits=ALL`, `EnableOnSubgraphs=0`
- Model: ONNX opset 14, FP16, fixed batch 128 and sequence 128
Contributor guide
Research direction
Start with the reproducer tests TestLateOnInferenceMatchesReferenceScores and TestLateOnCoreMLProviderMatrix in the linked seek commit, then read coreml_execution_provider.cc, builders/helper.cc, and builders/op_builder_factory.cc. Compare the static and non-static partition paths and the affected FP16 outputs on macOS 15 ARM64. Done means CoreML matches the CPU EP within tolerance or declines the partition when it cannot preserve ONNX semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100