aws / aws/amazon-sagemaker-examples

[Bug Report] SM Data Parallel with large models - bash command to execute the docker container build script does not work

Open
#2,045 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
- https://github.com/aws/amazon-sagemaker-examples/blob/master/training/distributed_training/pytorch/data_parallel/bert/pytorch_smdataparallel_bert_demo.ipynb
- https://github.com/aws/amazon-sagemaker-examples/blob/master/training/distributed_training/pytorch/data_parallel/maskrcnn/pytorch_smdataparallel_maskrcnn_demo.ipynb

**Describe the bug**
Ran the first 5 code cells for pushing the provided docker container with the image and tags:
```
image = "bert-smdataparallel-sagemaker" # Example: bert-smdataparallel-sagemaker
tag = "pt-1-6" # Example: pt1.6
```

The following cell does not complete the docker build:
```
%%time
! chmod +x build_and_push.sh; bash build_and_push.sh {region} {image} {tag}
```
returning the following error:
```
Provided region_name '{region}' doesn't match a supported format.

Provided region_name '{region}' doesn't match a supported format.
Error: Cannot perform an interactive login from a non TTY device
invalid argument "{image}" for "-t, --tag" flag: invalid reference format
See 'docker build --help'.
Error parsing reference: "{image}" is not a valid repository/tag: invalid reference format
invalid reference format
Error: Image build and push failed
CPU times: user 20.7 ms, sys: 6.68 ms, total: 27.4 ms
Wall time: 1.76 s
```

**To Reproduce**
Run the first 5 code cells of the notebook.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.