aws / aws/amazon-sagemaker-examples

Support PyTorch reduce_all in DDP using SMDDP so we can Parallelize Validation

Open
#3,967 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

Please support reduce_all using the Data Parallel DDP API for tensors while using the SMDPP backend so that we can run our validation runs in parallel rather than on a single rank.

`validation_loss = dist.all_reduce(loss, op=ReduceOp.AVG)`

Results in:

`RuntimeError: SMDDP does not support: ReduceOp`

For example, so we don't have to restrict validation to a single rank like this:

https://github.com/aws/amazon-sagemaker-examples/blob/c266495f4a4b8e9c65f288edad5f0729c5ca3959/training/distributed_training/pytorch/data_parallel/mnist/code/train_pytorch_smdataparallel_mnist.py#L255

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.