aws / aws/amazon-sagemaker-examples

Training Job Fails for Compression RL Notebook

Open
#1,474 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

In the notebook: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/reinforcement_learning/rl_network_compression_ray_custom/rl_network_compression_ray_custom.ipynb

Training job fails when executing the estimator:

`estimator.fit(cifar_inputs)`

It returns:

`
UnexpectedStatusException: Error for Training job rl-nnc-2020-08-xx-xx-xx-xx-xxx: Failed. Reason: AlgorithmError: ExecuteUserScriptError: Command "/usr/bin/python train-ray.py --rl.training.checkpoint_freq 1 --rl.training.stop.training_iteration 1 --s3_bucket sagemaker-us-west-2-xxxxxxxxxxxx`

To reproduce error:

1. Launch SM Notebook instance
2. Clone https://github.com/awslabs/amazon-sagemaker-examples.git
3. Open https://github.com/awslabs/amazon-sagemaker-examples/blob/master/reinforcement_learning/rl_network_compression_ray_custom/rl_network_compression_ray_custom.ipynb
4. Click "Cell" -> "Run All"

Contributor guide

Open the contributing guide

Research direction

Start with reinforcement_learning/rl_network_compression_ray_custom/rl_network_compression_ray_custom.ipynb and reproduce the failure by running all cells in an Amazon SageMaker Notebook instance. Inspect the estimator.fit(cifar_inputs) execution and the reported train-ray.py AlgorithmError. Done means the notebook completes training without the UnexpectedStatusException.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.