aws / aws/amazon-sagemaker-examples

cannot find fairseq image when creating container

Open
#1,701 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I am using fairseq_sagemaker_translate_en2fr notebook for training. When I am building a container for fairseq using command:
```
%%sh
chmod +x create_container.sh

./create_container.sh pytorch-fairseq
```
getting:

```
Getting from region us-east-1 and account 198247849549
Sending build context to Docker daemon 16.84MB
Step 1/16 : FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04
The push refers to repository [198247849549.dkr.ecr.us-east-1.amazonaws.com/pytorch-fairseq]
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://198247849549.dkr.ecr.us-east-1.amazonaws.com/v2/: dial tcp: lookup 198247849549.dkr.ecr.us-east-1.amazonaws.com on [::1]:53: read udp [::1]:51282->[::1]:53: read: connection refused
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://520713654638.dkr.ecr.us-east-1.amazonaws.com/v2/: dial tcp: lookup 520713654638.dkr.ecr.us-east-1.amazonaws.com on [::1]:53: read udp [::1]:58091->[::1]:53: read: connection refused
Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on [::1]:53: read udp [::1]:55097->[::1]:53: read: connection refused
Error response from daemon: No such image: pytorch-fairseq:latest
An image does not exist locally with the tag: 198247849549.dkr.ecr.us-east-1.amazonaws.com/pytorch-fairseq

---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
in
----> 1 get_ipython().run_cell_magic('sh', '', 'chmod +x create_container.sh \n\n./create_container.sh pytorch-fairseq\n')
~/anaconda2/envs/xyz/sagemaker/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
2380 with self.builtin_trap:
2381 args = (magic_arg_s, cell)
-> 2382 result = fn(*args, **kwargs)
2383 return result
2384
~/anaconda2/envs/xyz/sagemaker/lib/python3.7/site-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
140 else:
141 line = script
--> 142 return self.shebang(line, cell)
143
144 # write a basic docstring:
in shebang(self, line, cell)
~/anaconda2/envs/xyz/sagemaker/lib/python3.7/site-packages/IPython/core/magic.py in (f, *a, **k)
185 # but it's overkill for just that one bit of state.
186 def magic_deco(arg):
--> 187 call = lambda f, *a, **k: f(*a, **k)
188
189 if callable(arg):
~/anaconda2/envs/xyz/sagemaker/lib/python3.7/site-packages/IPython/core/magics/script.py in shebang(self, line, cell)
243 sys.stderr.flush()
244 if args.raise_error and p.returncode!=0:
--> 245 raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
246
247 def _run_script(self, p, cell, to_close):
CalledProcessError: Command 'b'chmod +x create_container.sh \n\n./create_container.sh pytorch-fairseq\n'' returned non-zero exit status 1.
```
Looks like it cannot find fairseq image. Any help here?

Contributor guide

Open the contributing guide

Research direction

Start with the fairseq_sagemaker_translate_en2fr notebook and create_container.sh, then inspect the Docker and ECR steps around the failed image lookup. Re-run the shown command while checking connectivity to the reported ECR and Docker registry endpoints. Done means the pytorch-fairseq image is built or found locally and the script completes its tagging and push steps.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, jupyter-notebook, python
Domain
cloud, devops, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.