aws / aws/amazon-sagemaker-examples
PyTorch SM DDP example fails when run in VPC
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/blob/master/training/distributed_training/pytorch/data_parallel/maskrcnn/pytorch_smdataparallel_maskrcnn_demo.ipynb
**Describe the bug**
I followed the steps to stage COCO and setup FSX linked with s3 and have been hitting a bug. It looks like this example uses Detectron2 in the backend and makes a call to its model_zoo which fails because the training job is running in a VPC. What is the workaround here?
[1,15]: File "/root/DeepLearningExamples/PyTorch/Segmentation/MaskRCNN/pytorch/maskrcnn_benchmark/utils/model_zoo.py", line 59, in cache_url
[1,15]:urllib.error.URLError:
A clear and concise description of what the bug is.
Attach error logs if available.
**To Reproduce**
Follow the steps outlined in the notebook.
Contributor guide
Research direction
Start by running the linked pytorch_smdataparallel_maskrcnn_demo.ipynb in the reported VPC setup and review the traceback at DeepLearningExamples/PyTorch/Segmentation/MaskRCNN/pytorch/maskrcnn_benchmark/utils/model_zoo.py, especially cache_url. Determine why the model_zoo download cannot connect from the training job, then document and verify a workable VPC-safe path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, pytorch
- Domain
- cloud, machine-learning, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100