deezer / deezer/spleeter

[Bug]Each training period: ConcatOp:Dimension of inputs should match

Open
#515 2 comments 0 reactions 0 assignees View on GitHub
bug invalid
Dominant language
Python
Stars
28.4k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

## Description

I'm a novice of deep learning. The training set prepared for 2stems.Sample_rate of training set is 8KHz,and the format of which is wav.
Each training period,Some errors were shown on the console that ConcatOp:Dimension of inputs should match:shape[0]=[*,*,2] vs. shape[1]=[*,*,1].
Details as follows:

## Step to reproduce

1. spleeter train --verbose -p path of json path of audio training set

## Output
gs\\atmb\\train\\118.80T_1001_0015_Cessna_CockpitVoice2\\vocals.wav' from 11.260
000000000002 to 31.26
INFO:spleeter:Loading audio b'K:\\Anaconda3\\Lib\\site-packages\\spleeter\\confi
gs\\atmb\\train\\118.80T_1001_0015_Cessna_CockpitVoice1\\vocals.wav' from 11.260
000000000002 to 31.26
WARNING:tensorflow:Entity > could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bu
g, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attac
h the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `expor
t AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bu
g, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attac
h the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bu
g, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attac
h the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `expor
t AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `expor
t AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bu
g, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attac
h the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `expor
t AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
INFO:spleeter:Audio data loaded successfully
INFO:spleeter:Audio data loaded successfully
INFO:spleeter:Loading audio b'K:\\Anaconda3\\Lib\\site-packages\\spleeter\\confi
gs\\atmb\\train\\118.80T_1001_0015_Cessna_CockpitVoice1\\other.wav' from 0.5 to
20.5
INFO:spleeter:Audio data loaded successfully
INFO:spleeter:Audio data loaded successfully
INFO:spleeter:Loading audio b'K:\\Anaconda3\\Lib\\site-packages\\spleeter\\confi
gs\\atmb\\train\\118.80T_1001_0015_Cessna_CockpitVoice2\\other.wav' from 0.5 to
20.5
INFO:spleeter:Loading audio b'K:\\Anaconda3\\Lib\\site-packages\\spleeter\\confi
gs\\atmb\\train\\118.80T_1001_0015_Cessna_CockpitVoice2\\other.wav' from 11.2600
00000000002 to 31.26
INFO:spleeter:Loading audio b'K:\\Anaconda3\\Lib\\site-packages\\spleeter\\confi
gs\\atmb\\train\\118.80T_1001_0015_Cessna_CockpitVoice1\\other.wav' from 11.2600
00000000002 to 31.26
WARNING:tensorflow:Entity > could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bug
, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach
the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `expor
t AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bug
, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach
the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bug
, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach
the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `expor
t AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `expor
t AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
INFO:spleeter:Audio data loaded successfully
WARNING:tensorflow:Entity > could not be transformed and will
be executed as-is. Please report this to the AutoGraph team. When filing the bug
, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach
the full output. Cause: Mangled names are not yet supported by AutoGraph
WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to th
e AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `expor
t AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Mangled names are
not yet supported by AutoGraph
INFO:spleeter:Audio data loaded successfully
INFO:spleeter:Audio data loaded successfully
INFO:spleeter:Audio data loaded successfully
Traceback (most recent call last):
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 1365, in _do_call
return fn(*args)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 1350, in _run_fn
target_list, run_metadata)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __
inference_Dataset_map_DatasetBuilder.random_time_crop_1250}} ConcatOp : Dimensio
ns of inputs should match: shape[0] = [858,1024,2] vs. shape[1] = [858,1024,1]
[[{{node concat}}]]
[[IteratorGetNext]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "K:\Anaconda3\Scripts\spleeter-script.py", line 10, in
sys.exit(entrypoint())
File "K:\Anaconda3\lib\site-packages\spleeter\__main__.py", line 54, in entryp
oint
main(sys.argv)
File "K:\Anaconda3\lib\site-packages\spleeter\__main__.py", line 46, in main
entrypoint(arguments, params)
File "K:\Anaconda3\lib\site-packages\spleeter\commands\train.py", line 98, in
entrypoint
evaluation_spec)
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\tra
ining.py", line 473, in train_and_evaluate
return executor.run()
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\tra
ining.py", line 613, in run
return self.run_local()
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\tra
ining.py", line 714, in run_local
saving_listeners=saving_listeners)
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\est
imator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\est
imator.py", line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\est
imator.py", line 1195, in _train_model_default
saving_listeners)
File "K:\Anaconda3\lib\site-packages\tensorflow_estimator\python\estimator\est
imator.py", line 1494, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored
_session.py", line 754, in run
run_metadata=run_metadata)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored
_session.py", line 1259, in run
run_metadata=run_metadata)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored
_session.py", line 1360, in run
raise six.reraise(*original_exc_info)
File "K:\Anaconda3\lib\site-packages\six.py", line 693, in reraise
raise value
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored
_session.py", line 1345, in run
return self._sess.run(*args, **kwargs)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored
_session.py", line 1418, in run
run_metadata=run_metadata)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\training\monitored
_session.py", line 1176, in run
return self._sess.run(*args, **kwargs)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 956, in run
run_metadata_ptr)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 1359, in _do_run
run_metadata)
File "K:\Anaconda3\lib\site-packages\tensorflow_core\python\client\session.py"
, line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: ConcatOp : Dimens
ions of inputs should match: shape[0] = [858,1024,2] vs. shape[1] = [858,1024,1]

[[node concat (defined at \lib\site-packages\tensorflow_core\python\fra
mework\ops.py:1748) ]]
[[IteratorGetNext]]

Original stack trace for 'concat':
File "\Scripts\spleeter-script.py", line 10, in
sys.exit(entrypoint())
File "\lib\site-packages\spleeter\__main__.py", line 54, in entrypoint
main(sys.argv)
File "\lib\site-packages\spleeter\__main__.py", line 46, in main
entrypoint(arguments, params)
File "\lib\site-packages\spleeter\commands\train.py", line 98, in entrypoint
evaluation_spec)
File "\lib\site-packages\tensorflow_estimator\python\estimator\training.py", l
ine 473, in train_and_evaluate
return executor.run()
File "\lib\site-packages\tensorflow_estimator\python\estimator\training.py", l
ine 613, in run
return self.run_local()
File "\lib\site-packages\tensorflow_estimator\python\estimator\training.py", l
ine 714, in run_local
saving_listeners=saving_listeners)
File "\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py",
line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py",
line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py",
line 1195, in _train_model_default
saving_listeners)
File "\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py",
line 1490, in _train_with_estimator_spec
log_step_count_steps=log_step_count_steps) as mon_sess:
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 584, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 1014, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 725, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 1207, in __init__
_WrappedSession.__init__(self, self._create_session())
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 1212, in _create_session
return self._sess_creator.create_session()
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 878, in create_session
self.tf_sess = self._session_creator.create_session()
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 638, in create_session
self._scaffold.finalize()
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 205, in finalize
default_ready_op)
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 287, in get_or_default
op = default_constructor()
File "\lib\site-packages\tensorflow_core\python\training\monitored_session.py"
, line 201, in default_ready_op
], 0)
File "\lib\site-packages\tensorflow_core\python\util\dispatch.py", line 180, i
n wrapper
return target(*args, **kwargs)
File "\lib\site-packages\tensorflow_core\python\ops\array_ops.py", line 1420,
in concat
return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
File "\lib\site-packages\tensorflow_core\python\ops\gen_array_ops.py", line 12
56, in concat_v2
"ConcatV2", values=values, axis=axis, name=name)
File "\lib\site-packages\tensorflow_core\python\framework\op_def_library.py",
line 794, in _apply_op_helper
op_def=op_def)
File "\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507
, in new_func
return func(*args, **kwargs)
File "\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357,
in create_op
attrs, op_def, compute_device)
File "\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426,
in _create_op_internal
op_def=op_def)
File "\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748,
in __init__
self._traceback = tf_stack.extract_stack()

## Environment

| OS | Windows 7 |
| Installation type | Conda |
| Hardware spec | CPU |
| spleeter 1.5.3 |
| tensorflow 1.15.0 |

## Additional context

Contributor guide

Open the contributing guide

Research direction

Reproduce the `spleeter train --verbose -p` command using the reported training set and begin with the traceback at `DatasetBuilder.random_time_crop`. Compare the reported audio inputs and formats, including the 8 kHz WAV set and the logged 44.1 kHz streams, against the failing shapes `[858,1024,2]` and `[858,1024,1]`. Done means training no longer fails with the ConcatOp dimension mismatch.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
audio-video-rtc, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.