awslabs / awslabs/sagemaker-debugger

Turn off debugger hooks in PyTorch?

Open
#401 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
165
Forks
82
PR merge metrics
No merged PRs in 30d

Description

Hi, I'm training with sagemaker using a custom docker image I created by extending the pytorch 1.6 training image with additional ssh settings for Horovod. I'm also using a custom estimator which I created by subclassing the pytorch estimator and adding a distribution parameter and configuration method.

When I launch single-node training everything works fine, but when I attempt to launch multi-node training, I get smdebug hook errors, even though I did not set any debugger rules. How can I turn off this functionality altogether?

> [1,10]: File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 717, in _call_impl
> [1,10]: hook = smd_utils.get_smdebug_hook()
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/torch/utils/smdebug.py", line 35, in get_smdebug_hook
> [1,10]: return smd.get_hook(create_if_not_exists=True)
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/smdebug/pytorch/singleton_utils.py", line 22, in get_hook
> [1,10]: create_if_not_exists=create_if_not_exists,
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/singleton_utils.py", line 60, in get_hook
> [1,10]: _create_hook(json_config_path, hook_class)
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/singleton_utils.py", line 28, in _create_hook
> [1,10]: hook = hook_class.create_from_json_file(json_file_path=json_config_path)
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/hook.py", line 294, in create_from_json_file
> [1,10]: return create_hook_from_json_config(cls, json_config_path=json_file_path)
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/json_config.py", line 243, in create_hook_from_json_config
> [1,10]: save_all=save_all,
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/smdebug/pytorch/hook.py", line 47, in __init__
> [1,10]: include_workers=include_workers,
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/hook.py", line 955, in __init__
> [1,10]: include_workers=include_workers,
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/hook.py", line 243, in __init__
> [1,10]: self._initialize_to_last_saved_state()
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/hook.py", line 251, in _initialize_to_last_saved_state
> [1,10]: self.state_store = StateStore()
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/state_store.py", line 42, in __init__
> [1,10]: self._read_states_file()
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/state_store.py", line 76, in _read_states_file
> [1,10]: parameters = json.load(json_data)
> [1,10]: File "/opt/conda/lib/python3.6/json/__init__.py", line 299, in load
> [1,10]: parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
> [1,10]: File "/opt/conda/lib/python3.6/json/__init__.py", line 354, in loads
> [1,10]: return _default_decoder.decode(s)
> [1,10]: File "/opt/conda/lib/python3.6/json/decoder.py", line 342, in decode
> [1,10]: raise JSONDecodeError("Extra data", s, end)
> [1,10]:json.decoder.JSONDecodeError: Extra data: line 1 column 1444 (char 1443)
> [1,10]:Error in atexit._run_exitfuncs:
> [1,10]:Traceback (most recent call last):
> [1,10]: File "/opt/conda/lib/python3.6/site-packages/smdebug/core/hook.py", line 961, in _cleanup
> [1,10]: if not self.exported_collections:
> [1,10]:AttributeError: 'Hook' object has no attribute 'exported_collections'

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.