microsoft / microsoft/onnxruntime
[Performance] CoreML EP: ~10x inference slowdown on iOS in 1.29 vs 1.28 (ConvNeXt-Tiny FP16)
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
After upgrading from ORT 1.28 to 1.29, every inference through the CoreML EP is ~10x slower on iOS (ARM64). Android NNAPI is unaffected on the same build. Downgrading to 1.28 immediately restores normal performance.
Model details:
- Architecture : ConvNeXt-Tiny (convnext_tiny.fb_in22k_ft_in1k via timm)
- Opset : 17
- Precision : FP16 weights (onnxconverter_common float16, keep_io_types=True — I/O stays FP32)
- Input shape : [1, 3, 224, 224] fixed (no dynamic axes)
- Export path : FP32 → ORT_ENABLE_EXTENDED graph optimization → FP16 conversion
The slowdown occurs on every inference call, not just the first — so it is not a CoreML model compilation/warm-up issue.
Workarounds attempted, none helped:
- ORT_DISABLE_TELEMETRY=1
- IntraOpNumThreads = 2, InterOpNumThreads = 1
- CoreMLFlags.COREML_FLAG_ENABLE_ON_SUBGRAPH
- session.disable_specified_optimizers = MatMulAddFusion (ref #32212)
ConvNeXt-Tiny does not use MatMul+Add patterns — it uses depthwise conv + LayerNorm + GELU. The #32212 fix does not apply here, which suggests the CoreML EP regression in 1.29 is broader than transformer/Gemm models.
### To reproduce
1. Export ConvNeXt-Tiny from timm to ONNX (opset 17, fixed shape [1,3,224,224])
2. Apply ORT graph optimization (ORT_ENABLE_EXTENDED)
3. Convert to FP16 using onnxconverter_common float16 with keep_io_types=True
4. Load the model on iOS via CoreML EP using ORT 1.29
5. Run inference repeatedly and measure per-call latency
6. Compare against ORT 1.28 with identical model and session options
### Urgency
This is a clear regression — ORT 1.28 works correctly, 1.29 does not. We have pinned back to 1.28 as a workaround and cannot upgrade until this is resolved.
### Platform
iOS
### OS Version
IOS 16.7.16
### ONNX Runtime Installation
Released Package
### ONNX Runtime Version or Commit ID
1.28 (working) → 1.29 (regressed)
### ONNX Runtime API
C#
### Architecture
ARM64
### Execution Provider
CoreML
### Execution Provider Library Version
_No response_
### Model File
ConvNeXt-Tiny FP16 ONNX, opset 17, fixed shape 1×3×224×224
### Is this a quantized model?
No
Contributor guide
Research direction
Reproduce the repeated-inference latency difference using ConvNeXt-Tiny FP16 with the fixed [1,3,224,224] input on iOS, comparing ORT 1.28 and 1.29 through the CoreML EP. Start by inspecting CoreML EP changes between those releases and benchmark per-call latency after warm-up. Done means identifying and correcting the 1.29 regression without relying on the listed workarounds, with the comparison reproducible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, ios, machine-learning
- Domain
- machine-learning, mobile-dev, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100