[Discussion] musdb18 training

Hello!

I am having trouble with training and I am trying to replicate the results in wiki like this #512 . I installed spleeter with pip, spleeter version 2.2.1. I downloaded musdb18.

I used this, spleeter train --verbose -p configs/musdb_config.json -d musdb18/
and the result is shown below.

INFO:tensorflow:Using config: {'_model_dir': 'musdb_model', '_tf_random_seed': 3, '_save_summary_steps': 5, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': gpu_options {
per_process_gpu_memory_fraction: 0.45
}
, '_keep_checkpoint_max': 2, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 10, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': ClusterSpec({}), '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
INFO:spleeter:Start model training
INFO:tensorflow:Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 1000 or save_checkpoints_secs None.
WARNING:tensorflow:From /home/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Apply unet for vocals_spectrogram
INFO:tensorflow:Apply unet for drums_spectrogram
INFO:tensorflow:Apply unet for bass_spectrogram
INFO:tensorflow:Apply unet for other_spectrogram
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Calling checkpoint listeners before saving checkpoint 0...
INFO:tensorflow:Saving checkpoints for 0 into musdb_model/model.ckpt.
INFO:tensorflow:Calling checkpoint listeners after saving checkpoint 0...
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Apply unet for vocals_spectrogram
INFO:tensorflow:Apply unet for drums_spectrogram
INFO:tensorflow:Apply unet for bass_spectrogram
INFO:tensorflow:Apply unet for other_spectrogram
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2021-04-18T12:34:55Z
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from musdb_model/model.ckpt-0
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Inference Time : 10.32122s
INFO:tensorflow:Finished evaluation at 2021-04-18-12:35:05
INFO:tensorflow:Saving dict for global step 0: absolute_difference = 0.0, bass_spectrogram = 0.0, drums_spectrogram = 0.0, global_step = 0, loss = 0.0, other_spectrogram = 0.0, vocals_spectrogram = 0.0
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 0: musdb_model/model.ckpt-0
WARNING:tensorflow:Training with estimator made no steps. Perhaps input is empty or misspecified.
INFO:tensorflow:Loss for final step: None.
INFO:spleeter:Model training done

Seems like I get zero training result. Is it safe to neglect the warnings?
I have been getting this result many times now. Even with my custom models.
Thank you in advance.

Start with the training command and configs/musdb_config.json, then compare the setup with wiki reference #512. Inspect the log's “input is empty or misspecified” warning and determine why training makes no steps. Done means the cause is documented or training produces actual steps and nonzero results.

Written by the indexing model from the issue text.