[Performance] CoreML EP: MatMulAddFusion forces Gemm weights inline into model.mil instead of weights/weight.bin (3.7 GB MIL, 17.3 GiB peak RSS)

Describe the issue

With default graph optimization, the CoreML EP (MLProgram format) writes essentially all model weights inline into model.mil as ASCII hex-float rather than into the binary weights file. Disabling exactly one optimizer — MatMulAddFusion — moves them back.

The failure is silent: a 7,269x larger MIL and a 7.5x larger peak RSS look like normal CoreML behaviour unless you compare against a run with the optimizer disabled.

Mechanism. ModelBuilder has two tensor-emit paths and only one can externalize. In onnxruntime/core/providers/coreml/builders/model_builder.cc at v1.29.0:

• line 799 — OnnxTensorToCoreMLTensor(tensor, *weights_file_writer_), which writes to weight.bin;
• line 718 — AddConstantImpl calls CreateTensorValue<float>(value, shape), which takes no StorageWriter and emits an inline immediatevalue.

MatMulAddFusion is a Level1 transformer, so it runs at every optimization level except ORT_DISABLE_ALL. It rewrites MatMul + Add into Gemm, and GemmOpBuilder re-emits the claimed B initializer through AddConstant — so every fused matmul weight takes the inline path. This matches the observed op counts (linear( 144 -> 0 as the Gemms disappear).

I have not built ONNX Runtime from source to step through it; the measurements below are firm and the mechanism is read from the source above. Happy to run further probes on this hardware.

On a 16 GB machine the fused path would likely OOM.

To reproduce

Load an fp32 transformer encoder through the CoreML EP with MLProgram format and static input shapes, once with default optimization and once with optimization.disable_specified_optimizers set to MatMulAddFusion. (Note that value is split on ;, not , — see #32211.) Then compare model.mil against weights/weight.bin in the compiled .mlmodelc.

Measured A/B with the same model.onnx hardlinked into both arms, so the weights are byte-identical and only the setting varies. Model: mxbai-embed-large-v1 (fp32 encoder, ~1.125 GiB of weights), batch 1, sequence length 512:

	fusion ON (default)	fusion OFF
model.mil	3,708,210,001 B	510,143 B
weights/weight.bin	3,407,168 B	1,211,375,936 B
peak RSS at load	17.334 GiB	2.323 GiB
warm load	83.93 s	0.60 s
linear( ops in MIL	144	0
matmul( ops in MIL	48	192

Weight conservation across the move: 1,207,968,768 B in, 1,207,965,813 B out — a 2,955 B residual on 1.125 GiB. Peak RSS replicated across independent compile+load cycles (17.334/17.334 and 2.323/2.285).

This is not a numerics or throughput tradeoff: with fusion off, throughput is unchanged to slightly better (n=200/arm, 30.13 -> 29.54 ms mean, faster at every percentile), and the output embeddings are bit-identical between arms.

Urgency

Not blocking — the workaround below is effective and we ship with it. Filing because the failure mode is silent and the memory cost is large enough to OOM smaller machines.

Platform

Mac

OS Version

macOS 15 (Darwin 25.6.0), Apple silicon

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

Measured on 1.22.0. The two code paths cited above are unchanged in v1.29.0.

ONNX Runtime API

C++ (via the Rust ort binding, dynamic loading)

Architecture

ARM64

Execution Provider

CoreML

Execution Provider Library Version

Built-in CoreML EP, MLProgram format, ComputeUnits::All, static input shapes.

Model File

mxbai-embed-large-v1, the public fp32 ONNX export (BERT-large-shaped encoder). Reproduces on any fp32 model with MatMul + Add pairs.

Is this a quantized model?

No

Start in onnxruntime/core/providers/coreml/builders/model_builder.cc, comparing AddConstantImpl and OnnxTensorToCoreMLTensor, then reproduce the optimization-on and MatMulAddFusion-disabled A/B run described above. Done means fused fp32 MatMul weights are externalized to weights/weight.bin rather than expanding model.mil, without changing the bit-identical outputs or fusion behavior.