apple / apple/coremltools

Diamond of Conv2DTranspose fails to convert

Open
#978 2 comments 0 reactions 0 assignees View on GitHub
bug tf2.x / tf.keras triaged
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🐞Describe the bug

"Diamond" keras model with Conv2DTranspose fails to convert. Note that TensorFlow can infer this model.

![image](https://user-images.githubusercontent.com/993053/97052925-e98f1480-154f-11eb-82d3-5b4d8214e961.png)

## Trace
If applicable, please paste the error trace.

```Running TensorFlow Graph Passes: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 5/5 [00:00<00:00, 67.97 passes/s]
Converting Frontend ==> MIL Ops: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 69/69 [00:00<00:00, 2411.42 ops/s]
Running MIL optimization passes: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 17/17 [00:00<00:00, 1940.17 passes/s]
Translating MIL ==> MLModel Ops: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 19/19 [00:00<00:00, 7695.23 ops/s]
/Users/thomaspeters/.virtualenvs/keras2coreml/lib/python3.7/site-packages/coremltools/models/model.py:119: RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "compiler error: Espresso exception: "Invalid blob shape": generic_elementwise_kernel: cannot broadcast [13, 391, 128, 1, 1] and [14, 392, 128, 1, 1]".
RuntimeWarning,
Traceback (most recent call last):
File "test2.py", line 28, in
print('COREML RESULT:', mlmodel.predict({'input_1': model_input})['Identity'].ravel())
File "/Users/thomaspeters/.virtualenvs/keras2coreml/lib/python3.7/site-packages/coremltools/models/model.py", line 367, in predict
raise self._framework_error
File "/Users/thomaspeters/.virtualenvs/keras2coreml/lib/python3.7/site-packages/coremltools/models/model.py", line 113, in _get_proxy_and_spec
return (_MLModelProxy(filename, use_cpu_only), specification, None)
RuntimeError: Error compiling model: "compiler error: Espresso exception: "Invalid blob shape": generic_elementwise_kernel: cannot broadcast [13, 391, 128, 1, 1] and [14, 392, 128, 1, 1]".
```

## To Reproduce

```
import coremltools
from tensorflow import keras
from tensorflow.keras.layers import Conv2DTranspose, Add
import numpy as np

out_ch = 128
input_shape = (1, 196, 7, 7)

inputs = keras.Input(shape=input_shape[1:])
x = Conv2DTranspose(out_ch, 3,
padding="same",
strides=(2,2),
input_shape=input_shape[1:],
output_padding=1,
use_bias=False)(inputs)
y = Conv2DTranspose(out_ch, 1, strides=(2,2), output_padding=1,use_bias=False)(inputs)
output = Add()([x,y])

model = keras.Model(inputs, output)

model_input = np.ones(input_shape)

print('KERAS RESULT:', np.array(model(model_input)).ravel())

mlmodel = coremltools.convert(model)
mlmodel.save('dummy.mlmodel')

print('COREML RESULT:', mlmodel.predict({'input_1': model_input})['Identity'].ravel())
```

## System environment:
- coremltools version: 4.0
- OS: MacOS
- macOS version: Catalina 10.15.7
- XCode version: Version 12.1 (12A7403)
- How you install python: system
- python version: 3.7.7
- any other relevant information:
- keras version: 2.4.3
- TensorFlow version: 2.2.0

## Additional context
This is a minimal reproducing example I could find of a problem we hit on a production model. For our production model, I was able to hack coremltools to get conversion to work by forcing an `output_shape` in coremltools/converters/mil/frontend/tensorflow/ops.py `def Conv2DBackpropInput`. I don't know how to actually fix this.

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.