aws / aws/amazon-sagemaker-examples
Support PyTorch reduce_all in DDP using SMDDP so we can Parallelize Validation
Open
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
Please support reduce_all using the Data Parallel DDP API for tensors while using the SMDPP backend so that we can run our validation runs in parallel rather than on a single rank.
`validation_loss = dist.all_reduce(loss, op=ReduceOp.AVG)`
Results in:
`RuntimeError: SMDDP does not support: ReduceOp`
For example, so we don't have to restrict validation to a single rank like this:
https://github.com/aws/amazon-sagemaker-examples/blob/c266495f4a4b8e9c65f288edad5f0729c5ca3959/training/distributed_training/pytorch/data_parallel/mnist/code/train_pytorch_smdataparallel_mnist.py#L255
Contributor guide
Assessment
This issue has not been assessed yet.