apple / apple/coremltools

Converted Pytroch model can't support flexible Input/Output Shape for ImageType

Open
#1,075 1 comment 4 reactions 0 assignees View on GitHub
bug Core ML Framework PyTorch (traced)
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🐞Describe the bug
- Possibly related to to https://github.com/apple/coremltools/issues/756, https://github.com/apple/coremltools/issues/880, https://github.com/apple/coremltools/issues/970, https://github.com/apple/coremltools/issues/992
- Encounter error when running an inference task with a converted mlmodel configured for flexible input + output shape.
- Model converted from PyTorch.
- No error during conversation, only runtime error.
- Verified bug on both XCode & CoreMLTools prediction API

## Trace/Error Message
# XCode
```
# Code
let output = try? model!.prediction(image: pixelBuffer!)

# Trace
[espresso] [Espresso::handle_ex_plan] exception=Espresso exception: "Invalid argument": generic_general_concat_kernel: axis out of bounds status=-6
[coreml] Failure dynamically resizing for sequence length.
[coreml] Failure in resetSizes.
```

# Using coremltools API
```
# Code
output = model.predict({'image': img})

# Trace
Traceback (most recent call last):
File "/Users/user/Documents/fast_nst/iphone_test.py", line 23, in
out_dict = model.predict({'image': img})
File "/Users/user/miniconda3/envs/NST/lib/python3.8/site-packages/coremltools/models/model.py", line 329, in predict
return self.__proxy__.predict(data, useCPUOnly)
RuntimeError: {
NSLocalizedDescription = "Failure dynamically resizing for sequence length.";
}
```

## To Reproduce [FB8985338 with Source File]
1. Convert a Pytroch model into CoreML Model & configure for flexible input/output [CODE BELOW]
2. Run prediction using coremltools API on the .mlmodel generated (In this case, make sure to test it with an image of size different from the one configured for the model).

- I attached the code file in FB8985338 that contains:
- convert.py (Configure the model & Convert the model)
- inference.py (Use coremltools API to perform prediction)
- Model Weight File
- Model Source File

- Below is the example code that performs the conversion & configuration for flexible input/output
```
# Configure output to be of ImaegType
def convert_multiarray_output_to_image(spec, feature_name, is_bgr=False):
from coremltools.proto import FeatureTypes_pb2 as ft
spec.description.output[0].type.multiArrayType.shape.append(3)
spec.description.output[0].type.multiArrayType.shape.append(HEIGHT)
spec.description.output[0].type.multiArrayType.shape.append(WIDTH)
for output in spec.description.output:
if output.name != feature_name:
continue
if output.type.WhichOneof('Type') != 'multiArrayType':
raise ValueError("%s is not a multiarray type" % output.name)
array_shape = tuple(output.type.multiArrayType.shape)
channels, height, width = array_shape
if channels == 1:
output.type.imageType.colorSpace = ft.ImageFeatureType.ColorSpace.Value('GRAYSCALE')
elif channels == 3:
if is_bgr:
output.type.imageType.colorSpace = ft.ImageFeatureType.ColorSpace.Value('BGR')
else:
output.type.imageType.colorSpace = ft.ImageFeatureType.ColorSpace.Value('RGB')
else:
raise ValueError("Channel Value %d not supported for image inputs" % channels)
output.type.imageType.width = width
output.type.imageType.height = height

# Configure spec to support flexible input & output image
def convert_flexible(spec):
img_size_ranges = flexible_shape_utils.NeuralNetworkImageSizeRange()
img_size_ranges.add_height_range((64, -1))
img_size_ranges.add_width_range((64, -1))
flexible_shape_utils.update_image_size_range(spec, feature_name=INPUT_NAME, size_range=img_size_ranges)
flexible_shape_utils.update_image_size_range(spec, feature_name=OUTPUT_NAME, size_range=img_size_ranges)

net = Load_Model().eval() # Set in Eval mode!
net.load_state_dict(torch.load(MODEL_WEIGHTS_PATH, map_location=torch.device(device)))
net = net.to(device)

example_input = torch.rand(1, 3, HEIGHT, WIDTH) # Figure out the proper size
traced_model = torch.jit.trace(net, example_input)

model_from_torch = ct.convert(traced_model, source="pytorch",
inputs=[ct.ImageType(name=INPUT_NAME, shape=example_input.shape)])
model_from_torch = ct.models.MLModel(model_spec)
model_from_torch.save(f"model.mlmodel")
```

## System environment (please complete the following information):
- coremltools version (e.g., 3.0b5): 4.0
- OS (e.g., MacOS, Linux): MacOS Big Sur
- macOS version (if applicable): 11.1
- XCode version (if applicable): 12.4
- How you install python (anaconda, virtualenv, system): miniconda
- python version (e.g. 3.7): 3.8.5
- any other relevant information:
- torch==1.6.0

## Additional context
1. Submitted bug report with source file at FB8985338.

Contributor guide

Open the contributing guide

Research direction

Start with the attached convert.py and inference.py, then review the flexible_shape_utils calls that update the input and output image ranges. Reproduce the failure using a converted PyTorch model and an image size different from the configured size through the CoreMLTools prediction API. Done means flexible image input and output shapes run without the reported dynamic resizing or axis errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.