aws / aws/amazon-sagemaker-examples
[Debugger] mxnet_spot_training/mxnet-spot-training-with-sagemakerdebugger.ipynb
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
Tested notebook: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-debugger/mxnet_spot_training/mxnet-spot-training-with-sagemakerdebugger.ipynb
The spot training job with Debugger fails:
```
FileNotFoundError: [Errno 2] No such file or directory: '/opt/ml/checkpoints/metadata.json.sagemaker-uploading'[2020-07-30 22:13:57.793 ip-10-0-92-158.us-east-2.compute.internal:16 INFO utils.py:25] The end of training job file will not be written for jobs running under SageMaker.2020-07-30 22:13:57,940 sagemaker-training-toolkit ERROR ExecuteUserScriptError:Command "/usr/local/bin/python3.6 mxnet_gluon_spot_training.py --batch-size 100 --checkpoint-path /opt/ml/checkpoints —epochs 5“
```
Contributor guide
Assessment
This issue has not been assessed yet.