apple / apple/coremltools

validator error on SplitV layer

Open
#1,043 0 comments 0 reactions 0 assignees View on GitHub
bug tf2.x / tf.keras
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🐞Describe the bug
I tried to convert Google SPICE Model (a pitch detection model) to a coreml model. Then I met a runtime Warning.
(The model can be downloaded from https://tfhub.dev/google/spice/2)

RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "Error reading protobuf spec. validator error: Layer 'stft/frame/split' of type 975 has 1 outputs but expects at least 2.". RuntimeWarning,

Layer 'stft/frame/split' is a **SplitV** layer, please see attached screenshots from netron app. (in additional context section)

## Trace
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing
(type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing
(type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing
(type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing
(type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing
(type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing
(type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Running TensorFlow Graph Passes: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 7/7 [00:00<00:00, 23.22 passes/s]
Converting Frontend ==> MIL Ops: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 720/720 [00:00<00:00, 820.04 ops/s]
Running MIL optimization passes: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 17/17 [00:00<00:00, 22.71 passes/s]
Translating MIL ==> MLModel Ops: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 768/768 [00:00<00:00, 2247.57 ops/s]
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/coremltools/models/model.py:119: RuntimeWarning: You will not be
able to run predict() on this Core ML model. Underlying exception message was: Error compiling model: "Error reading protobuf spec. validator e
rror: Layer 'stft/frame/split' of type 975 has 1 outputs but expects at least 2.".
RuntimeWarning,

## To Reproduce
- If a python script can reproduce the error, please paste the code snippet
```
# Imports requires for custom ops
from coremltools.converters.mil.mil.ops.defs._op_reqs import *
from coremltools.converters.mil.mil.types.symbolic import is_symbolic
from coremltools.converters.mil.mil import Builder as mb
from coremltools.converters.mil.frontend.tensorflow.tf_op_registry import register_tf_op
import coremltools as ct

@register_op(doc_str='Swift RFFT', is_custom_op=True)
class custom_rfft(Operation):
input_spec = InputSpec(
x = TensorInputType(),
fft_length = IntInputType(const=True, default=512),
)

bindings = { 'class_name' : 'SwiftRFFT',
'input_order' : ['x'],
'description' : "Swift RFFT"
}

def __init__(self, **kwargs):
super(custom_rfft, self).__init__(**kwargs)

def type_inference(self):
x_type = self.x.dtype
return types.tensor(x_type, [-1, 257])

@register_op(doc_str='Swift Real', is_custom_op=True)
class custom_real(Operation):
input_spec = InputSpec(
x = TensorInputType(),
)

bindings = { 'class_name' : 'SwiftReal',
'input_order' : ['x'],
'description' : "Swift Real"
}

def __init__(self, **kwargs):
super(custom_real, self).__init__(**kwargs)

def type_inference(self):
x_type = self.x.dtype
return types.tensor(x_type, [-1, 257])

@register_op(doc_str='Swift Imag', is_custom_op=True)
class custom_imag(Operation):
input_spec = InputSpec(
x = TensorInputType(),
)

bindings = { 'class_name' : 'SwiftImag',
'input_order' : ['x'],
'description' : "Swift Imag"
}

def __init__(self, **kwargs):
super(custom_imag, self).__init__(**kwargs)

def type_inference(self):
x_type = self.x.dtype
return types.tensor(x_type, [-1, 257])

@register_tf_op(tf_alias=['RFFT'], override=True)
def CustomRFFT(context, node):
x = context[node.inputs[0]]
fft_length = context[node.inputs[1]]
x = mb.custom_rfft(x=x, fft_length=fft_length, name=node.name)
context.add(node.name, x)

@register_tf_op(tf_alias=['Real'], override=True)
def CustomReal(context, node):
x = context[node.inputs[0]]
x = mb.custom_real(x=x, name=node.name)
context.add(node.name, x)

@register_tf_op(tf_alias=['Imag'], override=True)
def CustomImag(context, node):
x = context[node.inputs[0]]
x = mb.custom_imag(x=x, name=node.name)
context.add(node.name, x)

# the path to downloaded model from Tensorflow Hub
# The model can be downloaded from https://tfhub.dev/google/spice/2
mlmodel = ct.convert('./spice')
```

## System environment (please complete the following information):
- coremltools version (e.g., 4.0): 4.0
- macOS version (if applicable): 11.1
- XCode version (if applicable): 12.3
- How you install python (anaconda, virtualenv, system): installer from https://www.python.org/downloads/mac-osx/
- python version (e.g. 3.7): 3.6.8
- Tensorflow: 1.14

## Additional context
In netron app, 'stft/frame/split' layer is a SplitV layer.
In saved model:
![Screen Shot 0002-12-31 at 11 32 25](https://user-images.githubusercontent.com/32716/103393182-65af6500-4b5c-11eb-936d-b7c0f2129c90.jpg)

In its tflite model:
![Screen Shot 0002-12-31 at 11 32 07](https://user-images.githubusercontent.com/32716/103393217-90012280-4b5c-11eb-80d7-f647f7f13217.jpg)

Contributor guide

Open the contributing guide

Research direction

Start by downloading the Google SPICE SavedModel and reproducing the issue with ct.convert('./spice'). Inspect how the TensorFlow SplitV layer named stft/frame/split is represented during conversion and compare the reported output count with the validator error. Done means the model converts without the SplitV validation warning and can run predict().

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.