apple / apple/coremltools

Core ML Model performance far lower on iOS 17 vs iOS 16 (iOS 17 not using Neural Engine)

Open
#2,004 9 comments 3 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🐞Describing the bug

Hello,

I'm not sure if this is 100% `coremltools` related or an iOS level issue.

But I've found since iOS 17 release that none of my Core ML models running on devices with iOS 17 use the neural engine, thus resulting in far slower performance.

The following screenshots show the performance of the same model (a PyTorch computer vision model) on an iPhone SE 3rd gen and iPhone 13 Pro (both use the A15 Bionic).

### iOS 16 - iPhone SE 3rd Gen

iOS 16 uses the ANE and results in fast prediction, load and compilation times.

ios16-mlmodel-prediction-time

### iOS 17 - iPhone 13 Pro

iOS 17 doesn't seem to use the ANE, thus the prediction, load and compilation times are all slower.

ios17-mlmodel-prediction-time

## To Reproduce

The following is my code I'm using to export my PyTorch vision model.

I've used the same code for the past few months with sensational results on iOS 16.

```python
# Convert to Core ML using the Unified Conversion API
coreml_model = ct.convert(
model=traced_model,
inputs=[image_input],
outputs=[ct.TensorType(name="output")],
classifier_config=ct.ClassifierConfig(class_names),
convert_to="neuralnetwork",
# compute_precision=ct.precision.FLOAT16,
compute_units=ct.ComputeUnit.ALL
)
```

## System environment:
- Xcode version: 15.0
- coremltools version: 7.0.0
- OS (e.g. MacOS version or Linux type): Linux Ubuntu 20.04 (for exporting), macOS 13.6 (for testing on Xcode)
- Any other relevant version information (e.g. PyTorch or TensorFlow version): PyTorch 2.0

## Additional context
- This happens across `"neuralnetwork"` and `"mlprogram"` type models, neither use the ANE on iOS 17 but both use the ANE on iOS 16

If anyone has a similar experience, I'd love to hear more.

Otherwise, if I'm doing something wrong for the exporting of models for iOS 17+, please let me know.

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.