aws / aws/amazon-sagemaker-examples

[Bug Report] with Instance_type = "local_gpu", SageMaker unable to find PyTorch inference container image.

Open
#3,110 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

Similar issue to one reported in https://github.com/aws/sagemaker-python-sdk/issues/1099 and https://github.com/aws/sagemaker-python-sdk/issues/1105.

**Link to the notebook**
[Add the link to the notebook.](https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/pytorch_cnn_cifar10/pytorch_local_mode_cifar10.ipynb)

**Describe the bug**
1. Bug-1
py_version="py3", is missing from PyTorch estimator. Without it, training fails. Correct estimator:
cifar10_estimator = PyTorch(
entry_point="source/cifar10.py",
role=role,
framework_version="1.7.1",
py_version="py3",
instance_count=1,
instance_type=instance_type
)

2. Bug-2
if we set Instance_type = "local_gpu",
the deployment fails:
cifar10_predictor = cifar10_estimator.deploy(initial_instance_count=1, instance_type=instance_type)
CalledProcessError: Command '['docker', 'pull', '763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-inference:1.7.1-gpu-py3']' returned non-zero exit status 1.

Adding $!pip install -U sagemaker did not help

if we set Instance_type = "ml.p3.2xlarge", deployment succeeds.

**To reproduce**
A clear, step-by-step set of instructions to reproduce the bug.
Clone repo.
add py_version="py3" to the estimator.

**Logs**
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.

---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
in
2
3 #cifar10_predictor = cifar10_estimator.deploy(initial_instance_count=1, instance_type="ml.p3.2xlarge")
----> 4 cifar10_predictor = cifar10_estimator.deploy(initial_instance_count=1, instance_type=instance_type)

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/estimator.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, use_compiled_model, wait, model_name, kms_key, data_capture_config, tags, **kwargs)
959
960 model.name = model_name
--> 961
962 return model.deploy(
963 instance_type=instance_type,

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/model.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, **kwargs)
787
788 if instance_type and instance_type.startswith("ml.inf") and not self._is_compiled_model:
--> 789 LOGGER.warning(
790 "Your model is not compiled. Please compile your model before using Inferentia."
791 )

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/session.py in endpoint_from_production_variants(self, name, production_variants, tags, kms_key, wait, data_capture_config_dict)
3568 if kms_key:
3569 config_options["KmsKeyId"] = kms_key
-> 3570 if data_capture_config_dict is not None:
3571 config_options["DataCaptureConfig"] = data_capture_config_dict
3572

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/session.py in create_endpoint(self, endpoint_name, config_name, tags, wait)
3070 """
3071 LOGGER.info("Creating endpoint with name %s", endpoint_name)
-> 3072
3073 tags = tags or []
3074 tags = _append_project_tags(tags)

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/local/local_session.py in create_endpoint(self, EndpointName, EndpointConfigName, Tags)
346 endpoint = _LocalEndpoint(EndpointName, EndpointConfigName, Tags, self.sagemaker_session)
347 LocalSagemakerClient._endpoints[EndpointName] = endpoint
--> 348 endpoint.serve()
349
350 def update_endpoint(self, EndpointName, EndpointConfigName): # pylint: disable=unused-argument

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/local/entities.py in serve(self)
576 )
577 self.container.serve(
--> 578 self.primary_container["ModelDataUrl"], self.primary_container["Environment"]
579 )
580

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/local/image.py in serve(self, model_dir, environment)
285
286 if _ecr_login_if_needed(self.sagemaker_session.boto_session, self.image):
--> 287 _pull_image(self.image)
288
289 self._generate_compose_file(

~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/local/image.py in _pull_image(image)
1094 logger.info("docker command: %s", pull_image_command)
1095
-> 1096 subprocess.check_output(pull_image_command.split())
1097 logger.info("image pulled: %s", image)

~/anaconda3/envs/pytorch_p36/lib/python3.6/subprocess.py in check_output(timeout, *popenargs, **kwargs)
354
355 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
--> 356 **kwargs).stdout
357
358

~/anaconda3/envs/pytorch_p36/lib/python3.6/subprocess.py in run(input, timeout, check, *popenargs, **kwargs)
436 if check and retcode:
437 raise CalledProcessError(retcode, process.args,
--> 438 output=stdout, stderr=stderr)
439 return CompletedProcess(process.args, retcode, stdout, stderr)
440

CalledProcessError: Command '['docker', 'pull', '763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-inference:1.7.1-gpu-py3']' returned non-zero exit status 1.

Contributor guide

Open the contributing guide

Research direction

Start with sagemaker-python-sdk/pytorch_cnn_cifar10/pytorch_local_mode_cifar10.ipynb and run the documented reproduction using the PyTorch estimator and its deploy entry point. Compare the py_version="py3" case with deployment using instance_type="local_gpu", and inspect the logged Docker image pull failure. Done means the notebook's estimator setup and local GPU deployment complete successfully without the reported errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, jupyter-notebook, python, pytorch
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.