aws / aws/amazon-sagemaker-examples

PyTorch HPO example failing

Open
#1,378 3 comments 2 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

The [PyTorch HPO example](https://github.com/awslabs/amazon-sagemaker-examples/tree/master/hyperparameter_tuning/pytorch_mnist) does not work and fails with error:

Failure reason:
AlgorithmError: ExecuteUserScriptError: Command "/opt/conda/bin/python mnist.py --backend gloo --batch-size 256 --epochs 6 --lr 0.04926795600136902" Traceback (most recent call last): **File "mnist.py", line 5, i to n import sagemaker_containers ModuleNotFoundError: No module named 'sagemaker_containers'**

Problem is happening for PyTorch versions > 1.3.1 possibly due to https://github.com/aws/sagemaker-pytorch-training-toolkit/issues/189

Manually adding `!pip install sagemaker-containers` to the SM notebook also does not help. Checked for PyTorch 1.6.0. 1.5.0, 1.5.1

Contributor guide

Open the contributing guide

Research direction

Start in hyperparameter_tuning/pytorch_mnist and reproduce the PyTorch HPO example with the affected PyTorch versions, focusing on the mnist.py import shown in the traceback. Check the related sagemaker-pytorch-training-toolkit issue and verify that the example runs without the sagemaker_containers ModuleNotFoundError.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.