microsoft / microsoft/onnxruntime

[Performance] CoreML EP: ~10x inference slowdown on iOS in 1.29 vs 1.28 (ConvNeXt-Tiny FP16)

Open
#32,218 0 comments 0 reactions 0 assignees View on GitHub
api:CSharp ep:CoreML performance platform:mobile
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

After upgrading from ORT 1.28 to 1.29, every inference through the CoreML EP is ~10x slower on iOS (ARM64). Android NNAPI is unaffected on the same build. Downgrading to 1.28 immediately restores normal performance.

Model details:
- Architecture : ConvNeXt-Tiny (convnext_tiny.fb_in22k_ft_in1k via timm)
- Opset : 17
- Precision : FP16 weights (onnxconverter_common float16, keep_io_types=True — I/O stays FP32)
- Input shape : [1, 3, 224, 224] fixed (no dynamic axes)
- Export path : FP32 → ORT_ENABLE_EXTENDED graph optimization → FP16 conversion

The slowdown occurs on every inference call, not just the first — so it is not a CoreML model compilation/warm-up issue.

Workarounds attempted, none helped:
- ORT_DISABLE_TELEMETRY=1
- IntraOpNumThreads = 2, InterOpNumThreads = 1
- CoreMLFlags.COREML_FLAG_ENABLE_ON_SUBGRAPH
- session.disable_specified_optimizers = MatMulAddFusion (ref #32212)

ConvNeXt-Tiny does not use MatMul+Add patterns — it uses depthwise conv + LayerNorm + GELU. The #32212 fix does not apply here, which suggests the CoreML EP regression in 1.29 is broader than transformer/Gemm models.

### To reproduce

1. Export ConvNeXt-Tiny from timm to ONNX (opset 17, fixed shape [1,3,224,224])
2. Apply ORT graph optimization (ORT_ENABLE_EXTENDED)
3. Convert to FP16 using onnxconverter_common float16 with keep_io_types=True
4. Load the model on iOS via CoreML EP using ORT 1.29
5. Run inference repeatedly and measure per-call latency
6. Compare against ORT 1.28 with identical model and session options

### Urgency

This is a clear regression — ORT 1.28 works correctly, 1.29 does not. We have pinned back to 1.28 as a workaround and cannot upgrade until this is resolved.

### Platform

iOS

### OS Version

IOS 16.7.16

### ONNX Runtime Installation

Released Package

### ONNX Runtime Version or Commit ID

1.28 (working) → 1.29 (regressed)

### ONNX Runtime API

C#

### Architecture

ARM64

### Execution Provider

CoreML

### Execution Provider Library Version

_No response_

### Model File

ConvNeXt-Tiny FP16 ONNX, opset 17, fixed shape 1×3×224×224

### Is this a quantized model?

No

Contributor guide

Open the contributing guide

Research direction

Reproduce the repeated-inference latency difference using ConvNeXt-Tiny FP16 with the fixed [1,3,224,224] input on iOS, comparing ORT 1.28 and 1.29 through the CoreML EP. Start by inspecting CoreML EP changes between those releases and benchmark per-call latency after warm-up. Done means identifying and correcting the 1.29 regression without relying on the listed workarounds, with the comparison reproducible.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, ios, machine-learning
Domain
machine-learning, mobile-dev, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.