espnet / espnet/espnet_onnx

Failed to convert Espnet2 model using external (Fbank+Pitch) features to ONNX format

Open
#92 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
169
Forks
25
PR merge metrics
No merged PRs in 30d

Description

I simply test the espnet_onnx function by train the Espnet2 model using recipe "librispeech_100".
The ONNX conversion working in model train using default features (FBank), which is good.

But when I change the feature_type from default to "fbank_pitch", which will pre-generate fbank+pitch features using Kaldi extractor and use it in subsequent training and decode, error pop up when I using espnet_onnx to convert that trained model.

Can suggest if there is any problem and how to fix it ?

Command I am using is as below
` python3 -m espnet_onnx.export --model_type asr --tag conformer_ext_feature --input asr_conformer_lr2e-3_warmup15k_amp_nondeterministic_valid.acc.ave.zip `

And here is the error shown
`
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/__main__.py", line 91, in
m.export_from_zip(
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/export_asr.py", line 191, in export_from_zip
self.export(model, tag_name, quantize, optimize)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/export_asr.py", line 91, in export
self._export_encoder(enc_model, export_dir, verbose)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/export_asr.py", line 246, in _export_encoder
self._export_model(model, verbose, path)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/export_asr.py", line 226, in _export_model
torch.onnx.export(
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/onnx/__init__.py", line 350, in export
return utils.export(
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/onnx/utils.py", line 163, in export
_export(
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/onnx/utils.py", line 1074, in _export
graph, params_dict, torch_out = _model_to_graph(
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/onnx/utils.py", line 727, in _model_to_graph
graph, params, torch_out, module = _create_jit_graph(model, args)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/onnx/utils.py", line 602, in _create_jit_graph
graph, torch_out = _trace_and_get_graph_from_model(model, args)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/onnx/utils.py", line 517, in _trace_and_get_graph_from_model
trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/jit/_trace.py", line 1175, in _get_trace_graph
outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/jit/_trace.py", line 127, in forward
graph, out = torch._C._create_graph_by_tracing(
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/jit/_trace.py", line 118, in wrapper
outs.append(self.inner(*trace_inputs))
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/models/encoders/conformer.py", line 104, in forward
xs_pad, mask = self.embed(feats, mask)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/models/language_models/embed.py", line 74, in forward
return self.model(x, mask)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/models/language_models/subsampling.py", line 48, in forward
x = self.out(x.transpose(1, 2).contiguous().view(b, t, c * f))
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (24x4864 and 5120x256)
`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.