Faster cropping and scaling of images
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
## ❓Question
I am trying to predict some features of a very small object in a very large image. To make it manageable I therefore go for a 2-step approach that takes a full resolution crop of the object, simplified like this:
```python
import torch
from torchvision.models import mobilenet_v3_large
model_1 = mobilenet_v3_large(num_classes=2)
model_2 = mobilenet_v3_large()
image = torch.randn((1, 3, 3840, 2160))
small_image = image[:, :, 16:-16:17, 9:-9:17] # downscale
center_yx = model_1(small_image).int() # find object in image
crop = image[:, :, center_yx[0]-112: center_yx[0]+112, center_yx[1]-112: center_yx[1]+112] # crop
features = model_2(crop) # predict features
```
This works well except it is very slow when converted to CoreML/ML program. When testing the model performance through Xcode it says that about 75% of the time is spent on image cropping and downscaling (`ios18.slice_by_index`).
Is `slice_by_index` expected to be this slow? Are there any faster alternatives to scaling and cropping images? Or any recommend way of tackling the problem?
Contributor guide
Research direction
Start with the Python reproduction using mobilenet_v3_large and the converted Core ML model, then inspect the Xcode performance report for ios18.slice_by_index. The issue needs an explanation of whether cropping and downscaling cause the reported overhead and, if possible, a supported faster approach; no specific test or acceptance criterion is provided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ios, python, pytorch
- Domain
- machine-learning, mobile-dev, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100