aws / aws/amazon-sagemaker-examples
Training Job Fails for Compression RL Notebook
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
In the notebook: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/reinforcement_learning/rl_network_compression_ray_custom/rl_network_compression_ray_custom.ipynb
Training job fails when executing the estimator:
`estimator.fit(cifar_inputs)`
It returns:
`
UnexpectedStatusException: Error for Training job rl-nnc-2020-08-xx-xx-xx-xx-xxx: Failed. Reason: AlgorithmError: ExecuteUserScriptError: Command "/usr/bin/python train-ray.py --rl.training.checkpoint_freq 1 --rl.training.stop.training_iteration 1 --s3_bucket sagemaker-us-west-2-xxxxxxxxxxxx`
To reproduce error:
1. Launch SM Notebook instance
2. Clone https://github.com/awslabs/amazon-sagemaker-examples.git
3. Open https://github.com/awslabs/amazon-sagemaker-examples/blob/master/reinforcement_learning/rl_network_compression_ray_custom/rl_network_compression_ray_custom.ipynb
4. Click "Cell" -> "Run All"
Contributor guide
Research direction
Start with reinforcement_learning/rl_network_compression_ray_custom/rl_network_compression_ray_custom.ipynb and reproduce the failure by running all cells in an Amazon SageMaker Notebook instance. Inspect the estimator.fit(cifar_inputs) execution and the reported train-ray.py AlgorithmError. Done means the notebook completes training without the UnexpectedStatusException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100