`interpolate` MIL conversion fails when `mode="area"`
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
## 🐞Describe the bug
Just found an interesting bug:
```python
from coremltools import convert, TensorType
from torch import zeros
from torch.jit import trace
from torch.nn import Module
from torch.nn.functional import interpolate
class Model (Module):
def __init__(self):
super().__init__()
def forward (self, input):
return interpolate(input, size=None, scale_factor=(2.0, 2.0), mode="area", align_corners=None)
model = Model()
input = zeros(1, 3, 128, 128)
output = model(input)
print(output.shape)
scripted_model = trace(model, input)
coreml_model = convert(scripted_model, inputs=[TensorType(name="input", shape=input.shape)])
```
Results in:
```
Traceback (most recent call last):
File "test.py", line 21, in
coreml_model = convert(scripted_model, inputs=[TensorType(name="input", shape=input.shape)])
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/_converters_entry.py", line 352, in convert
mlmodel = mil_convert(
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/converter.py", line 183, in mil_convert
return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/converter.py", line 210, in _mil_convert
proto, mil_program = mil_convert_to_proto(
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/converter.py", line 273, in mil_convert_to_proto
prog = frontend_converter(model, **kwargs)
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/converter.py", line 105, in __call__
return load(*args, **kwargs)
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 47, in load
return _perform_torch_convert(converter, debug)
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/frontend/torch/load.py", line 84, in _perform_torch_convert
prog = converter.convert()
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/frontend/torch/converter.py", line 250, in convert
convert_nodes(self.context, self.graph)
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 89, in convert_nodes
add_op(context, node)
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/frontend/torch/ops.py", line 1270, in adaptive_avg_pool2d
avg_pool = mb.avg_pool(
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/mil/ops/registry.py", line 63, in add_op
return cls._add_op(op_cls, **kwargs)
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/mil/builder.py", line 191, in _add_op
new_op.type_value_inference()
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/mil/operation.py", line 240, in type_value_inference
output_types = self.type_inference()
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/mil/ops/defs/pool.py", line 65, in type_inference
D_out_shape = spatial_dimensions_out_shape(
File "/Users/yusuf/Library/Python/3.8/lib/python/site-packages/coremltools/converters/mil/mil/ops/defs/_utils.py", line 250, in spatial_dimensions_out_shape
out_shape.append(math.floor((input_shape[r] + pad[r] - effective_ks[r]) / strides[r] + 1))
```
Referring to:
https://github.com/apple/coremltools/blob/e3032bf28a5b46e398f1f99b9d4ae9e57a08ffdc/coremltools/converters/mil/mil/ops/defs/_utils.py#L247
If I add the following line right before the calculation that fails:
```python
strides = [1 if x == 0 else x for x in strides]
```
The the conversion doesn't result in an error, **but the model isn't runnable**:
```
RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: Pooling layer: 51 , stride cannot be 0"
```
I believe this error also causes problems with shape inference if there are further layers that use the output of `interpolate`. Still investigating.
## Trace
See above.
## To Reproduce
See above.
## System environment (please complete the following information):
- coremltools 5.2 (latest from `pip`)
- macOS 12.2.1
- XCode version 13.2.1 (13C100)
- Python (`brew`) 3.8.9
- PyTorch 1.9.0
Contributor guide
Research direction
Reproduce the Torch conversion with the `interpolate(..., mode="area")` example, then inspect `coremltools/converters/mil/frontend/torch/ops.py` at `adaptive_avg_pool2d`. Trace the resulting pooling shape inference through `coremltools/converters/mil/mil/ops/defs/pool.py` and `_utils.py` near the failing calculation. Done means conversion completes with valid shapes and the generated Core ML model is runnable without a zero-stride error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100