aws / aws/amazon-sagemaker-examples
[Bug Report] with Instance_type = "local_gpu", SageMaker unable to find PyTorch inference container image.
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
Similar issue to one reported in https://github.com/aws/sagemaker-python-sdk/issues/1099 and https://github.com/aws/sagemaker-python-sdk/issues/1105.
**Link to the notebook**
[Add the link to the notebook.](https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker-python-sdk/pytorch_cnn_cifar10/pytorch_local_mode_cifar10.ipynb)
**Describe the bug**
1. Bug-1
py_version="py3", is missing from PyTorch estimator. Without it, training fails. Correct estimator:
cifar10_estimator = PyTorch(
entry_point="source/cifar10.py",
role=role,
framework_version="1.7.1",
py_version="py3",
instance_count=1,
instance_type=instance_type
)
2. Bug-2
if we set Instance_type = "local_gpu",
the deployment fails:
cifar10_predictor = cifar10_estimator.deploy(initial_instance_count=1, instance_type=instance_type)
CalledProcessError: Command '['docker', 'pull', '763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-inference:1.7.1-gpu-py3']' returned non-zero exit status 1.
Adding $!pip install -U sagemaker did not help
if we set Instance_type = "ml.p3.2xlarge", deployment succeeds.
**To reproduce**
A clear, step-by-step set of instructions to reproduce the bug.
Clone repo.
add py_version="py3" to the estimator.
**Logs**
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
in
2
3 #cifar10_predictor = cifar10_estimator.deploy(initial_instance_count=1, instance_type="ml.p3.2xlarge")
----> 4 cifar10_predictor = cifar10_estimator.deploy(initial_instance_count=1, instance_type=instance_type)
~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/estimator.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, use_compiled_model, wait, model_name, kms_key, data_capture_config, tags, **kwargs)
959
960 model.name = model_name
--> 961
962 return model.deploy(
963 instance_type=instance_type,
~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/model.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, **kwargs)
787
788 if instance_type and instance_type.startswith("ml.inf") and not self._is_compiled_model:
--> 789 LOGGER.warning(
790 "Your model is not compiled. Please compile your model before using Inferentia."
791 )
~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/session.py in endpoint_from_production_variants(self, name, production_variants, tags, kms_key, wait, data_capture_config_dict)
3568 if kms_key:
3569 config_options["KmsKeyId"] = kms_key
-> 3570 if data_capture_config_dict is not None:
3571 config_options["DataCaptureConfig"] = data_capture_config_dict
3572
~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/session.py in create_endpoint(self, endpoint_name, config_name, tags, wait)
3070 """
3071 LOGGER.info("Creating endpoint with name %s", endpoint_name)
-> 3072
3073 tags = tags or []
3074 tags = _append_project_tags(tags)
~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/local/local_session.py in create_endpoint(self, EndpointName, EndpointConfigName, Tags)
346 endpoint = _LocalEndpoint(EndpointName, EndpointConfigName, Tags, self.sagemaker_session)
347 LocalSagemakerClient._endpoints[EndpointName] = endpoint
--> 348 endpoint.serve()
349
350 def update_endpoint(self, EndpointName, EndpointConfigName): # pylint: disable=unused-argument
~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/local/entities.py in serve(self)
576 )
577 self.container.serve(
--> 578 self.primary_container["ModelDataUrl"], self.primary_container["Environment"]
579 )
580
~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/local/image.py in serve(self, model_dir, environment)
285
286 if _ecr_login_if_needed(self.sagemaker_session.boto_session, self.image):
--> 287 _pull_image(self.image)
288
289 self._generate_compose_file(
~/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/sagemaker/local/image.py in _pull_image(image)
1094 logger.info("docker command: %s", pull_image_command)
1095
-> 1096 subprocess.check_output(pull_image_command.split())
1097 logger.info("image pulled: %s", image)
~/anaconda3/envs/pytorch_p36/lib/python3.6/subprocess.py in check_output(timeout, *popenargs, **kwargs)
354
355 return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
--> 356 **kwargs).stdout
357
358
~/anaconda3/envs/pytorch_p36/lib/python3.6/subprocess.py in run(input, timeout, check, *popenargs, **kwargs)
436 if check and retcode:
437 raise CalledProcessError(retcode, process.args,
--> 438 output=stdout, stderr=stderr)
439 return CompletedProcess(process.args, retcode, stdout, stderr)
440
CalledProcessError: Command '['docker', 'pull', '763104351884.dkr.ecr.us-west-2.amazonaws.com/pytorch-inference:1.7.1-gpu-py3']' returned non-zero exit status 1.
Contributor guide
Research direction
Start with sagemaker-python-sdk/pytorch_cnn_cifar10/pytorch_local_mode_cifar10.ipynb and run the documented reproduction using the PyTorch estimator and its deploy entry point. Compare the py_version="py3" case with deployment using instance_type="local_gpu", and inspect the logged Docker image pull failure. Done means the notebook's estimator setup and local GPU deployment complete successfully without the reported errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, jupyter-notebook, python, pytorch
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100