aws / aws/amazon-sagemaker-examples

[Debugger] mxnet_spot_training/mxnet-spot-training-with-sagemakerdebugger.ipynb

Open
#1,518 0 comments 0 reactions 1 assignee Claimed by @hongshanli23 View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

Tested notebook: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-debugger/mxnet_spot_training/mxnet-spot-training-with-sagemakerdebugger.ipynb

The spot training job with Debugger fails:
```
FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/checkpoints/metadata.json.sagemaker-uploading'[2020-07-30 22:13:57.793 ip-10-0-92-158.us-east-2.compute.internal:16 INFO utils.py:25] The end of training job file will not be written for jobs running under SageMaker.2020-07-30 22:13:57,940 sagemaker-training-toolkit ERROR ExecuteUserScriptError:Command "/usr/local/bin/python3.6 mxnet_gluon_spot_training.py --batch-size 100 --checkpoint-path /opt/ml/checkpoints —epochs 5“
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.