apple / apple/coremltools

TensorflowTTS Keras model fails to export.

Open
#1,768 0 comments 1 reaction 0 assignees View on GitHub
bug tf2.x / tf.keras
Dominant language
Python
Stars
5.4k
Forks
850
Avg merge
4d 5h
Merged PRs (30d)
10

Description

## 🐞Describing the bug
Attempting to convert some of the text to speech models at https://github.com/TensorSpeech/TensorFlowTTS

The models work correctly under tensorflowlite/coreml but fail to directly export using CoreML Tools. I've included the stack trace and example code. You will need to install TensorFlowTTS python package.

## Stack Trace
```
Running TensorFlow Graph Passes: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 6/6 [00:00<00:00, 11.82 passes/s]
Converting TF Frontend ==> MIL Ops: 76%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Š | 1198/1585 [00:00<00:00, 6877.63 ops/s]
Traceback (most recent call last):
File "convert_tts.py", line 17, in
model = ct.convert(fastspeech2_convert,
File "Python/3.9/lib/python/site-packages/coremltools/converters/_converters_entry.py", line 444, in convert
mlmodel = mil_convert(
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/converter.py", line 187, in mil_convert
return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/converter.py", line 211, in _mil_convert
proto, mil_program = mil_convert_to_proto(
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/converter.py", line 281, in mil_convert_to_proto
prog = frontend_converter(model, **kwargs)
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/converter.py", line 99, in __call__
return tf2_loader.load()
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 82, in load
program = self._program_from_tf_ssa()
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 210, in _program_from_tf_ssa
return converter.convert()
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 474, in convert
self.convert_main_graph(prog, graph)
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 397, in convert_main_graph
outputs = convert_graph(self.context, graph, self.output_names)
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py", line 191, in convert_graph
add_op(context, node)
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/frontend/tensorflow/ops.py", line 1262, in RealDiv
y = mb.cast(x=context[node.inputs[1]], dtype="fp32")
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/mil/ops/registry.py", line 176, in add_op
return cls._add_op(op_cls_to_add, **kwargs)
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/mil/builder.py", line 182, in _add_op
new_op.type_value_inference()
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/mil/operation.py", line 256, in type_value_inference
output_vals = self._auto_val(output_types)
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/mil/operation.py", line 380, in _auto_val
vals = self.value_inference()
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/mil/operation.py", line 113, in wrapper
return func(self)
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/mil/ops/defs/iOS15/elementwise_unary.py", line 870, in value_inference
return self.get_cast_value(self.x, self.dtype.val)
File "Python/3.9/lib/python/site-packages/coremltools/converters/mil/mil/ops/defs/iOS15/elementwise_unary.py", line 896, in get_cast_value
return input_var.val.astype(dtype=type_map[dtype_val])
AttributeError: 'float' object has no attribute 'astype'

```

## To Reproduce

You will need to install TensorFlowTTS python package from https://github.com/TensorSpeech/TensorFlowTTS

- Please add a minimal code example that can reproduce the error when running it.
```
from tensorflow_tts.inference import AutoConfig
from tensorflow_tts.inference import TFAutoModel
import coremltools as ct

if __name__ == "__main__":
fastspeech2_config = AutoConfig.from_pretrained("tensorspeech/tts-fastspeech2-ljspeech-en")
fastspeech2_convert = TFAutoModel.from_pretrained(
config=fastspeech2_config,
name="fastspeech2",
enable_tflite_convertible=True,
)
print(fastspeech2_convert.summary())
model = ct.convert(fastspeech2_convert,
source='tensorflow',
convert_to='mlprogram',
minimum_deployment_target=ct.target.iOS16)
```

```
Model: "fastspeech2"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embeddings (TFFastSpeechEmb multiple 844032
eddings)

encoder (TFFastSpeechEncode multiple 11814400
r)

length_regulator (TFFastSpe multiple 0
echLengthRegulator)

decoder (TFFastSpeechDecode multiple 12601216
r)

mel_before (Dense) multiple 30800

postnet (TFTacotronPostnet) multiple 4352400

f0_predictor (TFFastSpeechV multiple 493313
ariantPredictor)

energy_predictor (TFFastSpe multiple 493313
echVariantPredictor)

duration_predictor (TFFastS multiple 493313
peechVariantPredictor)

f0_embeddings (Conv1D) multiple 3840

dropout_32 (Dropout) multiple 0

energy_embeddings (Conv1D) multiple 3840

dropout_33 (Dropout) multiple 0

=================================================================
Total params: 31,130,467
Trainable params: 29,552,579
Non-trainable params: 1,577,888
_________________________________________________________________

```

## System environment (please complete the following information):
- coremltools version: 6.2
- OS (e.g. MacOS version or Linux type): MacOS 13.2
- Any other relevant version information (e.g. PyTorch or TensorFlow version): Python 3.9, Tensorflow 2.10

Contributor guide

Open the contributing guide

Research direction

Start with the minimal TensorFlowTTS conversion example and inspect the TensorFlow converter path in frontend/tensorflow/ops.py, especially RealDiv, along with the cast handling in mil/ops/defs/iOS15/elementwise_unary.py. Done means the supplied FastSpeech2 conversion no longer raises the reported AttributeError during Core ML conversion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.