apple / apple/coremltools

Faster cropping and scaling of images

Open
#2,472 3 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## ❓Question
I am trying to predict some features of a very small object in a very large image. To make it manageable I therefore go for a 2-step approach that takes a full resolution crop of the object, simplified like this:

```python
import torch
from torchvision.models import mobilenet_v3_large

model_1 = mobilenet_v3_large(num_classes=2)
model_2 = mobilenet_v3_large()
image = torch.randn((1, 3, 3840, 2160))

small_image = image[:, :, 16:-16:17, 9:-9:17] # downscale
center_yx = model_1(small_image).int() # find object in image
crop = image[:, :, center_yx[0]-112: center_yx[0]+112, center_yx[1]-112: center_yx[1]+112] # crop
features = model_2(crop) # predict features
```

This works well except it is very slow when converted to CoreML/ML program. When testing the model performance through Xcode it says that about 75% of the time is spent on image cropping and downscaling (`ios18.slice_by_index`).

Is `slice_by_index` expected to be this slow? Are there any faster alternatives to scaling and cropping images? Or any recommend way of tackling the problem?

Contributor guide

Open the contributing guide

Research direction

Start with the Python reproduction using mobilenet_v3_large and the converted Core ML model, then inspect the Xcode performance report for ios18.slice_by_index. The issue needs an explanation of whether cropping and downscaling cause the reported overhead and, if possible, a supported faster approach; no specific test or acceptance criterion is provided.

Written by the indexing model from the issue text.

Assessment

Tech stack
ios, python, pytorch
Domain
machine-learning, mobile-dev, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.