aws / aws/amazon-sagemaker-examples

rl_roboschool_ray_automatic_model_tuning.ipynb fails to start hyperparameter tuning jobs

Open
#1,290 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

Hi Team,
can I ask you please to help me resolve issue which I faced where executing ' rl_roboschool_ray_automatic_model_tuning.ipynb' described below?
I am executing the code provided by AWS sagemaker without any change.

At the same point, I executed successfully 'rl_roboschool_ray.ipynb'. Both locally and by submitting a processing job as well.
However, when executing rl_roboschool_ray_automatic_model_tuning.ipynb I get the following error:
Failure reason
AlgorithmError: framework error: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/sagemaker_containers/_trainer.py", line 84, in train entrypoint() File "/usr/local/lib/python3.6/dist-packages/sagemaker_tensorflow_container/training.py", line 206, in main model_dir = _model_dir_with_training_job(hyperparameters.get('model_dir'), env.job_name) File "/usr/local/lib/python3.6/dist-packages/sagemaker_tensorflow_container/training.py", line 189, in _model_dir_with_training_job if model_dir.startswith('/opt/ml'): AttributeError: 'NoneType' object has no attribute 'startswith' 'NoneType' object has no attribute 'startswith'

The hyperparameter tuning job fails before it reaches entrypoint script.
I have no idea how I can further debug the above described issue.

Contributor guide

Open the contributing guide

Research direction

Start by running rl_roboschool_ray_automatic_model_tuning.ipynb and compare it with the successful rl_roboschool_ray.ipynb. Trace how the hyperparameter tuning job supplies model_dir and confirm that the job starts and reaches the entrypoint without the NoneType startswith error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python, tensorflow
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.