aws / aws/amazon-sagemaker-examples
Failed to install required packages
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
In the notebook: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_batch_transform/tensorflow_cifar-10_with_inference_script/tensorflow-serving-cifar10-python-sdk.ipynb
The step when you deploy to an endpoint with an EI accelerator:
`predictor = estimator_hvd.deploy(initial_instance_count=1, instance_type='ml.m5.xlarge', accelerator_type='ml.eia1.medium')`
fails with:
`UnexpectedStatusException: Error hosting endpoint dist-cifar10-tf-2020-08-xx-xx-xx-xx-xxx: Failed. Reason: The primary container,accelerator for production variant AllTraffic did not pass the ping health check. Please check CloudWatch logs for this endpoint..`
The last few outputs in CloudWatch log shows:
```
You are using pip version 8.1.1, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
ERROR:__main__:failed to install required packages, exiting.
INFO:__main__:stopping services
Traceback (most recent call last): File "/sagemaker/serve.py", line 211, in _setup_gunicorn subprocess.check_call(pip_install_cmd.split()) File "/usr/lib/python3.5/subprocess.py", line 581, in check_call raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pip3', 'install', '-r', '/opt/ml/model/code/requirements.txt']' returned non-zero exit status 1
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/sagemaker/serve.py", line 377, in ServiceManager().start() File "/sagemaker/serve.py", line 344, in start self._setup_gunicorn() File "/sagemaker/serve.py", line 214, in _setup_gunicorn self._stop() File "/sagemaker/serve.py", line 292, in _stop os.kill(self._nginx.pid, signal.SIGQUIT)
AttributeError: 'NoneType' object has no attribute 'pid'
```
To reproduce error:
1. Launch SM Notebook instance
2. Clone https://github.com/awslabs/amazon-sagemaker-examples.git
3. Open https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_batch_transform/tensorflow_cifar-10_with_inference_script/tensorflow-serving-cifar10-python-sdk.ipynb
4. Click "Cell" -> "Run All"
Contributor guide
Research direction
Open sagemaker_batch_transform/tensorflow_cifar-10_with_inference_script/tensorflow-serving-cifar10-python-sdk.ipynb and reproduce the failure with Run All. Inspect the CloudWatch logs and the /opt/ml/model/code/requirements.txt installation invoked by /sagemaker/serve.py. Done means the required packages install successfully and the deployed EI endpoint passes its ping health check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python, tensorflow
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100