Anyway to pass custom preprocess logic?
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
## ❓Question
Hi, I was wondering if there is anyway to include custom preprocessing logic during the model conversion process.
Regarding preprocesing, I could not see anything besides very basic, predefined preprocessing calls like below:
```
scale = 1/(0.226*255.0)
bias = [- 0.485/(0.229) , - 0.456/(0.224), - 0.406/(0.225)]
image_input = ct.ImageType(name="input_1",
shape=example_input.shape,
scale=scale, bias=bias)
```
What I would like to acheive is image padding as a way to work around this issue #1957. Because things break down when flexible shape is enabled, I was thinking that maybe if I pad the images which all differ in size, to very large, fixed size (say 2048, 2048) during runtime, and do inference on that (and later unpad it), I can live with the fixed size requirements, eventhough it's not really optimal.
Yes, I know that it would be totally possible to just write the preprocessing in the app side, but if I can embed the preprocessing to the model itself using simple Python, it would make my life much easier. Unfortunately, adding the pad logic inside the model definition fails:
```
class CoreMLaMa(torch.nn.Module):
def __init__(self, lama):
super(CoreMLaMa, self).__init__()
self.lama = lama
def autopad(self, x):
"""some custom logic to pad images"""
pass
def forward(self, image, mask):
"""Things crash before even reaching autopad""""
image = self.autopad(image)
mask = self.autopad(mask)
normalized_mask = ((mask > 0) * 1).byte()
lama_out = self.lama(image, normalized_mask)
output = torch.clamp(lama_out * 255, min=0, max=255)
return output
```
Because before entering the forward padding logic, there is dimension check happening at `model.predict()` very first, which throws `RuntimeError`. Is there anyway to make the preprocess logic kick in before the dimension check?
I saw some post talking about `proto` files for preprocessing, but this doesn't seem like a conventional way of preprocessing.
Is there anyway to make this work?
Contributor guide
Research direction
Start by tracing the model conversion path and the dimension check reported to occur in model.predict(), then review the existing ImageType preprocessing options and the mentioned proto-based preprocessing approach. Done would mean determining whether custom padding and unpadding can run before dimension validation, or documenting that preprocessing must remain in the application.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100