aws / aws/amazon-sagemaker-examples

[Bug Report] pytorch_flores_batch_transform - cannot build image

Open
#4,447 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
[Add the link to the notebook](https://github.com/aws/amazon-sagemaker-examples/tree/main/sagemaker_batch_transform/pytorch_flores_batch_transform)

**Describe the bug**
Building the container fails with the following error:
Step 1/23 : FROM nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04
manifest for nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04 not found: manifest unknown: manifest unknown

**To reproduce**
Run the notebook [sagemaker_batch_transform_torchserve.ipynb](https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker_batch_transform/pytorch_flores_batch_transform/sagemaker_batch_transform_torchserve.ipynb)

**Logs**
here is the redacted output of the cell that's titled 'Build a custom container':
```
Login Succeeded
Sending build context to Docker daemon 5.632kB
Step 1/23 : FROM nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04
Using default tag: latest
The push refers to repository [AWSACCOUNTID.dkr.ecr.ap-southeast-2.amazonaws.com/flores-torchserve-sagemaker]
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

manifest for nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04 not found: manifest unknown: manifest unknown
Error response from daemon: No such image: flores-torchserve-sagemaker:latest
An image does not exist locally with the tag: AWSACCOUNTID.dkr.ecr.ap-southeast-2.amazonaws.com/flores-torchserve-sagemaker
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
in
----> 1 get_ipython().run_cell_magic('sh', '', '\ncontainer_name=flores-torchserve-sagemaker\naccount=$(aws sts get-caller-identity --query Account --output text)\n\n# Get the region defined in the current configuration (default to us-west-2 if none defined)\nregion=$(aws configure get region)\n# region=${region:-us-west-2}\n\nfullname="${account}.dkr.ecr.${region}.amazonaws.com/${container_name}"\n\n# If the repository doesn\'t exist in ECR, create it.\naws ecr describe-repositories --repository-names "${container_name}" > /dev/null 2>&1\nif [ $? -ne 0 ]\nthen\n aws ecr create-repository --repository-name "${container_name}" > /dev/null\nfi\n\n# Get the login command from ECR and execute it directly\n$(aws ecr get-login --region ${region} --no-include-email)\n\n# Build the docker image locally with the image name and then push it to ECR\n# with the full name.\ndocker build -t ${container_name} docker-flo/\ndocker tag ${container_name} ${fullname}\n\ndocker push ${fullname}\n')

~/anaconda3/envs/python3/lib/python3.6/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
2369 with self.builtin_trap:
2370 args = (magic_arg_s, cell)
-> 2371 result = fn(*args, **kwargs)
2372 return result
2373

~/anaconda3/envs/python3/lib/python3.6/site-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
140 else:
141 line = script
--> 142 return self.shebang(line, cell)
143
144 # write a basic docstring:

in shebang(self, line, cell)

~/anaconda3/envs/python3/lib/python3.6/site-packages/IPython/core/magic.py in (f, *a, **k)
185 # but it's overkill for just that one bit of state.
186 def magic_deco(arg):
--> 187 call = lambda f, *a, **k: f(*a, **k)
188
189 if callable(arg):

~/anaconda3/envs/python3/lib/python3.6/site-packages/IPython/core/magics/script.py in shebang(self, line, cell)
243 sys.stderr.flush()
244 if args.raise_error and p.returncode!=0:
--> 245 raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
246
247 def _run_script(self, p, cell, to_close):

CalledProcessError: Command 'b'\ncontainer_name=flores-torchserve-sagemaker\naccount=$(aws sts get-caller-identity --query Account --output text)\n\n# Get the region defined in the current configuration (default to us-west-2 if none defined)\nregion=$(aws configure get region)\n# region=${region:-us-west-2}\n\nfullname="${account}.dkr.ecr.${region}.amazonaws.com/${container_name}"\n\n# If the repository doesn\'t exist in ECR, create it.\naws ecr describe-repositories --repository-names "${container_name}" > /dev/null 2>&1\nif [ $? -ne 0 ]\nthen\n aws ecr create-repository --repository-name "${container_name}" > /dev/null\nfi\n\n# Get the login command from ECR and execute it directly\n$(aws ecr get-login --region ${region} --no-include-email)\n\n# Build the docker image locally with the image name and then push it to ECR\n# with the full name.\ndocker build -t ${container_name} docker-flo/\ndocker tag ${container_name} ${fullname}\n\ndocker push ${fullname}\n'' returned non-zero exit status 1.

```

Contributor guide

Open the contributing guide

Research direction

Start with sagemaker_batch_transform/pytorch_flores_batch_transform/sagemaker_batch_transform_torchserve.ipynb and inspect the docker-flo/ build context, especially the base image referenced by the failing Docker build. Rerun the notebook's “Build a custom container” cell after the change; done means the image builds, is tagged, and can be pushed without the manifest or missing-image errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, jupyter-notebook, pytorch
Domain
devops, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.