apple / apple/coremltools

Anyway to pass custom preprocess logic?

Open
#1,959 2 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## ❓Question

Hi, I was wondering if there is anyway to include custom preprocessing logic during the model conversion process.

Regarding preprocesing, I could not see anything besides very basic, predefined preprocessing calls like below:

```
scale = 1/(0.226*255.0)
bias = [- 0.485/(0.229) , - 0.456/(0.224), - 0.406/(0.225)]

image_input = ct.ImageType(name="input_1",
shape=example_input.shape,
scale=scale, bias=bias)
```

What I would like to acheive is image padding as a way to work around this issue #1957. Because things break down when flexible shape is enabled, I was thinking that maybe if I pad the images which all differ in size, to very large, fixed size (say 2048, 2048) during runtime, and do inference on that (and later unpad it), I can live with the fixed size requirements, eventhough it's not really optimal.

Yes, I know that it would be totally possible to just write the preprocessing in the app side, but if I can embed the preprocessing to the model itself using simple Python, it would make my life much easier. Unfortunately, adding the pad logic inside the model definition fails:

```
class CoreMLaMa(torch.nn.Module):
def __init__(self, lama):
super(CoreMLaMa, self).__init__()
self.lama = lama

def autopad(self, x):
"""some custom logic to pad images"""
pass

def forward(self, image, mask):
"""Things crash before even reaching autopad""""
image = self.autopad(image)
mask = self.autopad(mask)

normalized_mask = ((mask > 0) * 1).byte()
lama_out = self.lama(image, normalized_mask)
output = torch.clamp(lama_out * 255, min=0, max=255)
return output

```
Because before entering the forward padding logic, there is dimension check happening at `model.predict()` very first, which throws `RuntimeError`. Is there anyway to make the preprocess logic kick in before the dimension check?

I saw some post talking about `proto` files for preprocessing, but this doesn't seem like a conventional way of preprocessing.

Is there anyway to make this work?

Contributor guide

Open the contributing guide

Research direction

Start by tracing the model conversion path and the dimension check reported to occur in model.predict(), then review the existing ImageType preprocessing options and the mentioned proto-based preprocessing approach. Done would mean determining whether custom padding and unpadding can run before dimension validation, or documenting that preprocessing must remain in the application.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.