aws / aws/amazon-sagemaker-examples

PyTorch SM DDP example fails when run in VPC

Open
#2,536 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/master/training/distributed_training/pytorch/data_parallel/maskrcnn/pytorch_smdataparallel_maskrcnn_demo.ipynb

**Describe the bug**
I followed the steps to stage COCO and setup FSX linked with s3 and have been hitting a bug. It looks like this example uses Detectron2 in the backend and makes a call to its model_zoo which fails because the training job is running in a VPC. What is the workaround here?

[1,15]: File "/root/DeepLearningExamples/PyTorch/Segmentation/MaskRCNN/pytorch/maskrcnn_benchmark/utils/model_zoo.py", line 59, in cache_url
[1,15]:urllib.error.URLError:

A clear and concise description of what the bug is.
Attach error logs if available.

**To Reproduce**
Follow the steps outlined in the notebook.

Contributor guide

Open the contributing guide

Research direction

Start by running the linked pytorch_smdataparallel_maskrcnn_demo.ipynb in the reported VPC setup and review the traceback at DeepLearningExamples/PyTorch/Segmentation/MaskRCNN/pytorch/maskrcnn_benchmark/utils/model_zoo.py, especially cache_url. Determine why the model_zoo download cannot connect from the training job, then document and verify a workable VPC-safe path.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, pytorch
Domain
cloud, machine-learning, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.