[Bug] Some GPU Training related error
- Dominant language
- Python
- Stars
- 28.4k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
## Step to reproduce
1. Installed using `pip`
2. Ran `spleeter train -p configs\hi_config.json -d D:/hi`
3. Got an error see output
## Output
*new output*
INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Apply unet for vocals_spectrogram
WARNING:tensorflow:From c:\program files\python37\lib\site-packages\tensorflow_core\python\ops\resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
INFO:tensorflow:Apply unet for piano_spectrogram
INFO:tensorflow:Apply unet for drums_spectrogram
INFO:tensorflow:Apply unet for bass_spectrogram
INFO:tensorflow:Apply unet for other_spectrogram
INFO:tensorflow:Apply unet for guitar_spectrogram
INFO:tensorflow:Apply unet for humperc_spectrogram
INFO:tensorflow:Apply unet for keys_spectrogram
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from hi\model.ckpt-0
Traceback (most recent call last):
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\client\session.py", line 1365, in _do_call
return fn(*args)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\client\session.py", line 1350, in _run_fn
target_list, run_metadata)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\client\session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key batch_normalization_72/beta not found in checkpoint
[[{{node save/RestoreV2}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 1290, in restore
{self.saver_def.filename_tensor_name: save_path})
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\client\session.py", line 956, in run
run_metadata_ptr)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\client\session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\client\session.py", line 1359, in _do_run
run_metadata)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\client\session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key batch_normalization_72/beta not found in checkpoint
[[node save/RestoreV2 (defined at c:\program files\python37\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]
Original stack trace for 'save/RestoreV2':
File "c:\program files\python37\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\program files\python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Program Files\Python37\Scripts\spleeter.exe\__main__.py", line 9, in
sys.exit(entrypoint())
File "c:\program files\python37\lib\site-packages\spleeter\__main__.py", line 54, in entrypoint
main(sys.argv)
File "c:\program files\python37\lib\site-packages\spleeter\__main__.py", line 46, in main
entrypoint(arguments, params)
File "c:\program files\python37\lib\site-packages\spleeter\commands\train.py", line 98, in entrypoint
evaluation_spec)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
return executor.run()
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
return self.run_local()
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1195, in _train_model_default
saving_listeners)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1490, in _train_with_estimator_spec
log_step_count_steps=log_step_count_steps) as mon_sess:
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 584, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1014, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 725, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1207, in __init__
_WrappedSession.__init__(self, self._create_session())
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1212, in _create_session
return self._sess_creator.create_session()
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 878, in create_session
self.tf_sess = self._session_creator.create_session()
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 638, in create_session
self._scaffold.finalize()
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 237, in finalize
self._saver.build()
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 840, in build
self._build(self._filename, build_save=True, build_restore=True)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 878, in _build
build_restore=build_restore)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 502, in _build_internal
restore_sequentially, reshape)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 381, in _AddShardedRestoreOps
name="restore_shard"))
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\ops\gen_io_ops.py", line 1696, in restore_v2
name=name)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 1300, in restore
names_to_keys = object_graph_key_mapping(save_path)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 1618, in object_graph_key_mapping
object_graph_string = reader.get_tensor(trackable.OBJECT_GRAPH_PROTO_KEY)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 915, in get_tensor
return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\program files\python37\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\program files\python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Program Files\Python37\Scripts\spleeter.exe\__main__.py", line 9, in
File "c:\program files\python37\lib\site-packages\spleeter\__main__.py", line 54, in entrypoint
main(sys.argv)
File "c:\program files\python37\lib\site-packages\spleeter\__main__.py", line 46, in main
entrypoint(arguments, params)
File "c:\program files\python37\lib\site-packages\spleeter\commands\train.py", line 98, in entrypoint
evaluation_spec)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
return executor.run()
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
return self.run_local()
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1195, in _train_model_default
saving_listeners)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1490, in _train_with_estimator_spec
log_step_count_steps=log_step_count_steps) as mon_sess:
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 584, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1014, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 725, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1207, in __init__
_WrappedSession.__init__(self, self._create_session())
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1212, in _create_session
return self._sess_creator.create_session()
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 878, in create_session
self.tf_sess = self._session_creator.create_session()
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 647, in create_session
init_fn=self._scaffold.init_fn)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\session_manager.py", line 290, in prepare_session
config=config)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\session_manager.py", line 220, in _restore_checkpoint
saver.restore(sess, ckpt.model_checkpoint_path)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 1306, in restore
err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Key batch_normalization_72/beta not found in checkpoint
[[node save/RestoreV2 (defined at c:\program files\python37\lib\site-packages\tensorflow_core\python\framework\ops.py:1748) ]]
Original stack trace for 'save/RestoreV2':
File "c:\program files\python37\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\program files\python37\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Program Files\Python37\Scripts\spleeter.exe\__main__.py", line 9, in
sys.exit(entrypoint())
File "c:\program files\python37\lib\site-packages\spleeter\__main__.py", line 54, in entrypoint
main(sys.argv)
File "c:\program files\python37\lib\site-packages\spleeter\__main__.py", line 46, in main
entrypoint(arguments, params)
File "c:\program files\python37\lib\site-packages\spleeter\commands\train.py", line 98, in entrypoint
evaluation_spec)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
return executor.run()
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
return self.run_local()
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1161, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1195, in _train_model_default
saving_listeners)
File "c:\program files\python37\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1490, in _train_with_estimator_spec
log_step_count_steps=log_step_count_steps) as mon_sess:
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 584, in MonitoredTrainingSession
stop_grace_period_secs=stop_grace_period_secs)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1014, in __init__
stop_grace_period_secs=stop_grace_period_secs)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 725, in __init__
self._sess = _RecoverableSession(self._coordinated_creator)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1207, in __init__
_WrappedSession.__init__(self, self._create_session())
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 1212, in _create_session
return self._sess_creator.create_session()
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 878, in create_session
self.tf_sess = self._session_creator.create_session()
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 638, in create_session
self._scaffold.finalize()
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\monitored_session.py", line 237, in finalize
self._saver.build()
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 840, in build
self._build(self._filename, build_save=True, build_restore=True)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 878, in _build
build_restore=build_restore)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 502, in _build_internal
restore_sequentially, reshape)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 381, in _AddShardedRestoreOps
name="restore_shard"))
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\training\saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\ops\gen_io_ops.py", line 1696, in restore_v2
name=name)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "c:\program files\python37\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()
## Environment
| ----------------- | ------------------------------- |
| OS | Windows 10 |
| Installation type| pip |
| RAM available | 32gb |
| Hardware spec | Ryzen 5 3600x/GTX 1060 6gb|
Contributor guide
Research direction
Reproduce the reported `spleeter train -p configs\hi_config.json -d D:/hi` command and start at `spleeter/commands/train.py`, whose entrypoint reaches TensorFlow Estimator training. Inspect the checkpoint restore failure for the missing `batch_normalization_72/beta` key and verify that training can restore the checkpoint without the reported NotFoundError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100