aws / aws/amazon-sagemaker-examples
Distributed training error using MXNet Gluon examples
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
I've tried run examples of MXNet Gluon scripts and notebooks ([sagemaker-python-sdk/mxnet_gluon_sentiment](https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/mxnet_gluon_sentiment/mxnet_sentiment_analysis_with_gluon.ipynb) and [sagemaker-python-sdk/mxnet_gluon_mnist](https://github.com/aws/amazon-sagemaker-examples/tree/master/sagemaker-python-sdk/mxnet_gluon_mnist)) with distributed training, and when the default framework version 1.6.0 (or 1.7.0, 1.8.0) is configured, the job failed with errors below:
```
[Epoch 1] Training: accuracy=0.909658
[Epoch 1] Validation: accuracy=0.812799
Vocabulary saved to "%s" /opt/ml/model/vocab.json
terminate called without an active exception
2020-12-03 03:58:54,540 sagemaker-training-toolkit ERROR ExecuteUserScriptError:
Command "/usr/local/bin/python3.6 sentiment.py --batch-size 8 --embedding-size 50 --epochs 2 --learning-rate 0.01 --log-interval 1000"
```
And change framework version to 1.4.1 would workaround this issue, SageMaker version is 2.* (2.1.0, 2.16.4, 2.17.0). Would like to know what is the cause and how to fix this using newer version like 1.6.0 and later.
Thanks.
Contributor guide
Research direction
Start by reproducing the failure in sagemaker-python-sdk/mxnet_gluon_sentiment and sagemaker-python-sdk/mxnet_gluon_mnist with framework versions 1.6.0 and later, comparing against 1.4.1. Trace the distributed training error from the example entry points and document or implement a verified fix for the newer versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100