aws / aws/amazon-sagemaker-examples
rl_roboschool_ray_automatic_model_tuning.ipynb fails to start hyperparameter tuning jobs
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
Hi Team,
can I ask you please to help me resolve issue which I faced where executing ' rl_roboschool_ray_automatic_model_tuning.ipynb' described below?
I am executing the code provided by AWS sagemaker without any change.
At the same point, I executed successfully 'rl_roboschool_ray.ipynb'. Both locally and by submitting a processing job as well.
However, when executing rl_roboschool_ray_automatic_model_tuning.ipynb I get the following error:
Failure reason
AlgorithmError: framework error: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/sagemaker_containers/_trainer.py", line 84, in train entrypoint() File "/usr/local/lib/python3.6/dist-packages/sagemaker_tensorflow_container/training.py", line 206, in main model_dir = _model_dir_with_training_job(hyperparameters.get('model_dir'), env.job_name) File "/usr/local/lib/python3.6/dist-packages/sagemaker_tensorflow_container/training.py", line 189, in _model_dir_with_training_job if model_dir.startswith('/opt/ml'): AttributeError: 'NoneType' object has no attribute 'startswith' 'NoneType' object has no attribute 'startswith'
The hyperparameter tuning job fails before it reaches entrypoint script.
I have no idea how I can further debug the above described issue.
Contributor guide
Research direction
Start by running rl_roboschool_ray_automatic_model_tuning.ipynb and compare it with the successful rl_roboschool_ray.ipynb. Trace how the hyperparameter tuning job supplies model_dir and confirm that the job starts and reaches the entrypoint without the NoneType startswith error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python, tensorflow
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100