apple / apple/coremltools

Conversion from tensorflow with ImageProjectiveTransformV2 op is not working.

Open
#1,426 7 comments 0 reactions 0 assignees View on GitHub
bug tf2.x / tf.keras triaged
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🐞Describe the bug
When you have an [ImageProjectiveTransformV2](https://www.tensorflow.org/versions/r2.6/api_docs/python/tf/raw_ops/ImageProjectiveTransformV2) op in tensorflow model and try to convert it to coreml then it raises the following exception. Is seems that `affine` ops does not support the "unnormalized" `coordinates_mode`. Do you plan to fix it in the near future? Affine transformation is quite commonly used in computer vision problems nowadays and it would be really nice to use it during conversion from tensorflow 🙏

## Trace
```
Traceback (most recent call last):
File "/Users/maciej3031/projects/coreml.py", line 257, in
mlmodel = ct.convert(model,
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/_converters_entry.py", line 352, in convert
mlmodel = mil_convert(
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 183, in mil_convert
return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 210, in _mil_convert
proto, mil_program = mil_convert_to_proto(
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 273, in mil_convert_to_proto
prog = frontend_converter(model, **kwargs)
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 95, in __call__
return tf2_loader.load()
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 84, in load
program = self._program_from_tf_ssa()
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 200, in _program_from_tf_ssa
return converter.convert()
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 401, in convert
self.convert_main_graph(prog, graph)
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 330, in convert_main_graph
outputs = convert_graph(self.context, graph, self.outputs)
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py", line 189, in convert_graph
add_op(context, node)
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/ops.py", line 1132, in ImageProjectiveTransformV2
x = mb.affine(
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/mil/ops/registry.py", line 63, in add_op
return cls._add_op(op_cls, **kwargs)
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/mil/builder.py", line 191, in _add_op
new_op.type_value_inference()
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/mil/operation.py", line 240, in type_value_inference
output_types = self.type_inference()
File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/mil/ops/defs/image_resizing.py", line 125, in type_inference
raise NotImplementedError(
NotImplementedError: input "coordinates_mode" to the "affine" not implemented. Got "unnormalized"
```

## Minimal code to Reproduce
```
import os
import numpy as np
import tensorflow as tf
import coremltools as ct
from PIL import Image

img01 = np.random.rand(256, 256, 3).astype(np.float32)
img0255 = (img01*255).astype(np.uint8)
img_pil = Image.fromarray(img0255)

class ProblemLayer(tf.keras.layers.Layer):
def __init__(self, **kwargs):
super(ProblemLayer, self).__init__(**kwargs)

def call(self, image):
B, H, W = tf.shape(image)[0], tf.shape(image)[1], tf.shape(image)[2]
proj_mat = tf.convert_to_tensor([[1.22, 0.09, -39.61, -0.09, 1.22, -6.17, 0.0, 0.0]], dtype=tf.float32)
proj_mat = tf.repeat(proj_mat, B, 0)
image = tf.raw_ops.ImageProjectiveTransformV2(
images=image,
transforms=proj_mat,
output_shape=[H, W],
interpolation='BILINEAR',
fill_mode='CONSTANT')
return image

inp = tf.keras.layers.Input(shape=(256, 256, 3), name="combined_input")
x = ProblemLayer()(inp)
model = tf.keras.models.Model(inputs=inp, outputs=x)

mlmodel = ct.convert(model,
inputs=[ct.ImageType(shape=(1, 256, 256, 3), scale=1.0/255.0)],
convert_to="mlprogram")

pred_coreml = mlmodel.predict({'combined_input': img_pil})
```

## System environment
- coremltools 5.2.0
- MacOS 12.2.1
- conda 4.11.0
- python 3.9.7
- tensorflow 2.6.0

## Additional context
Apart from this bug, do you plan to add support for [ImageProjectiveTransformV3](https://www.tensorflow.org/versions/r2.6/api_docs/python/tf/raw_ops/ImageProjectiveTransformV3)?

Contributor guide

Open the contributing guide

Research direction

Start by running the minimal TensorFlow reproduction and reading coremltools/converters/mil/frontend/tensorflow/ops.py at ImageProjectiveTransformV2, along with mil/ops/defs/image_resizing.py where the exception is raised. Trace how the unnormalized coordinates_mode reaches affine and check the existing conversion tests. Done means the supplied model converts successfully, with ImageProjectiveTransformV3 support clarified or covered if included in scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
machine-learning, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.