aws / aws/amazon-sagemaker-examples

Failed to install required packages

Open
#1,470 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

In the notebook: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_batch_transform/tensorflow_cifar-10_with_inference_script/tensorflow-serving-cifar10-python-sdk.ipynb

The step when you deploy to an endpoint with an EI accelerator:

`predictor = estimator_hvd.deploy(initial_instance_count=1, instance_type='ml.m5.xlarge', accelerator_type='ml.eia1.medium')`

fails with:

`UnexpectedStatusException: Error hosting endpoint dist-cifar10-tf-2020-08-xx-xx-xx-xx-xxx: Failed. Reason: The primary container,accelerator for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint..`

The last few outputs in CloudWatch log shows:

```
You are using pip version 8.1.1, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
ERROR:__main__:failed to install required packages, exiting.
INFO:__main__:stopping services
Traceback (most recent call last): File "/sagemaker/serve.py", line 211, in _setup_gunicorn subprocess.check_call(pip_install_cmd.split()) File "/usr/lib/python3.5/subprocess.py", line 581, in check_call raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip3', 'install', '-r', '/opt/ml/model/code/requirements.txt']' returned non-zero exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/sagemaker/serve.py", line 377, in ServiceManager().start() File "/sagemaker/serve.py", line 344, in start self._setup_gunicorn() File "/sagemaker/serve.py", line 214, in _setup_gunicorn self._stop() File "/sagemaker/serve.py", line 292, in _stop os.kill(self._nginx.pid, signal.SIGQUIT)
AttributeError: 'NoneType' object has no attribute 'pid'
```

To reproduce error:
1. Launch SM Notebook instance
2. Clone https://github.com/awslabs/amazon-sagemaker-examples.git
3. Open https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_batch_transform/tensorflow_cifar-10_with_inference_script/tensorflow-serving-cifar10-python-sdk.ipynb
4. Click "Cell" -> "Run All"

Contributor guide

Open the contributing guide

Research direction

Open sagemaker_batch_transform/tensorflow_cifar-10_with_inference_script/tensorflow-serving-cifar10-python-sdk.ipynb and reproduce the failure with Run All. Inspect the CloudWatch logs and the /opt/ml/model/code/requirements.txt installation invoked by /sagemaker/serve.py. Done means the required packages install successfully and the deployed EI endpoint passes its ping health check.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python, tensorflow
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.