awslabs / awslabs/sagemaker-debugger

TensorBoardOutputConfig/Sagemaker Debugger does not behave as documented

Open
#284 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
165
Forks
82
PR merge metrics
No merged PRs in 30d

Description

I have a working pytorch training script, which runs on my local machine. I'm trying to set it up to run on Sagemaker.
I want to save data to TensorBoard, get this uploaded into S3, and visualize via TensorBoard, like the documentation says here: https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_debugger.html#capture-real-time-tensorboard-data-from-the-debugging-hook

As mentioned, this works perfectl when running on my dev machine.
On Sagemaker, I do this:
```
tensorboard_output_config = TensorBoardOutputConfig(
s3_output_path=f's3://{args.s3_bucket}/tensorboard/{args.noise}',
container_local_output_path=f'/tb-logs'
)
```
then run it using:
` job = PyTorch(..., tensorboard_output_config=tensorboard_output_config, ...)`

Sagemaker creates the TensorBoard folder on S3, and seems to be logging everything I need. However, it is trying to log bunch of other things which I haven't specified. When the training starts, I see this in the log:
```
[2020-06-29 13:44:20.083 algo-1:52 INFO json_config.py:90] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.
[2020-06-29 13:44:20.084 algo-1:52 INFO hook.py:236] Saving to /opt/ml/output/tensors
[2020-06-29 13:44:20.084 algo-1:52 INFO state_store.py:67] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.
[2020-06-29 13:44:20.100 algo-1:52 INFO hook.py:376] Monitoring the collections: losses
[2020-06-29 13:44:20.100 algo-1:52 INFO hook.py:437] Hook is writing from the hook with pid: 52
```

Sagemaker is trying to log the losses from my networks, because they are all added into TensorBoard, with each of the losses having one value.
On TensorBoard, for every run, there are multiple directories and files. One of them contains all the information that I am logging myself. All the others are created my Sagemaker; they are clogging the UI and IMHO should not be there.

How can I turn this off?

Contributor guide

Open the contributing guide

Research direction

Start with the SageMaker Debugger documentation linked in the issue and the TensorBoardOutputConfig/PyTorch entry points. Inspect how /opt/ml/input/config/debughookconfig.json enables the losses collection, then verify that a configuration can leave only the user's TensorBoard data in S3 without the extra loss directories and files.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.