espnet / espnet/espnet_onnx

Error during export of joint_conformer_fastspeech2_hifigan model

Open
#106 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
169
Forks
25
PR merge metrics
No merged PRs in 30d

Description

I am trying to export a joint_conformer_fastspeech2_hifigan model to be loaded as an ONNX model. The code I used:

```python
from espnet_onnx.export import TTSModelExport
m = TTSModelExport()
m.export(text2speech, 'custom_model', quantize=True)
```

I also tried the above with a pre-trained model:
```python
from espnet_onnx.export import TTSModelExport
tag_name = 'kan-bayashi/ljspeech_joint_train_conformer_fastspeech2_hifigan'
m = TTSModelExport()
m.export_from_pretrained(tag_name)
```

But in both cases I got the same error below:
```python
Traceback (most recent call last):
File "/Users/vigourav/development/tts_bitbucket/tts_inference/inference_api.py", line 853, in
m.export(text2speech, 'custom_model', quantize=True)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/espnet_onnx/export/tts/export_tts.py", line 56, in export
self._export_tts(tts_model, export_dir, verbose)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/espnet_onnx/export/tts/export_tts.py", line 173, in _export_tts
self._export_model(model, verbose, path)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/espnet_onnx/export/tts/export_tts.py", line 154, in _export_model
torch.onnx.export(
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/onnx/utils.py", line 516, in export
_export(
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/onnx/utils.py", line 1613, in _export
graph, params_dict, torch_out = _model_to_graph(
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/onnx/utils.py", line 1135, in _model_to_graph
graph, params, torch_out, module = _create_jit_graph(model, args)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/onnx/utils.py", line 1011, in _create_jit_graph
graph, torch_out = _trace_and_get_graph_from_model(model, args)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/onnx/utils.py", line 915, in _trace_and_get_graph_from_model
trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/jit/_trace.py", line 1296, in _get_trace_graph
outs = ONNXTracedModule(
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/jit/_trace.py", line 138, in forward
graph, out = torch._C._create_graph_by_tracing(
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/jit/_trace.py", line 129, in wrapper
outs.append(self.inner(*trace_inputs))
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _slow_forward
result = self.forward(*input, **kwargs)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/espnet_onnx/export/tts/models/tts_models/fastspeech2.py", line 146, in forward
_, outs, d_outs, p_outs, e_outs = self._forward(
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/espnet_onnx/export/tts/models/tts_models/fastspeech2.py", line 195, in _forward
p_outs = self.pitch_predictor(hs, d_masks.unsqueeze(-1))
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _slow_forward
result = self.forward(*input, **kwargs)
File "/Users/vigourav/anaconda3/envs/oracle_espnet/lib/python3.9/site-packages/espnet2/tts/fastspeech2/variance_predictor.py", line 84, in forward
xs = xs.masked_fill(x_masks, 0.0)
RuntimeError: masked_fill_ only supports boolean masks, but got mask with dtype float
```

Is Joint Fastspeech2 hifigan model not supported yet? If not, by when can we expect it?

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce both export paths from the issue, then inspect espnet_onnx/export/tts/export_tts.py and espnet_onnx/export/tts/models/tts_models/fastspeech2.py around the failing export call. Trace the mask passed to espnet2/tts/fastspeech2/variance_predictor.py and compare the behavior for the joint_conformer_fastspeech2_hifigan model. Done means the supported export path no longer raises this mask-type error, or the model's unsupported status is clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.