add support for distributed backends to specify default backend in multiple backends format
- Dominant language
- Python
- Stars
- 103k
- Forks
- 29.5k
- PR merge metrics
- PR metrics pending
Description
### 🚀 The feature, motivation and pitch
Currently a dictionary exists for in-tree and custom device types to specify default backend to be used with `init_process_group(backend=None)`.
This is defined in `torch.distributed.distributed_c10d.py::Backend.default_device_backend_map`.
It would be useful for developers to specify a default backend in the colon format like: `cpu:gloo,custom_device:custom_backend` for the following reason:
Distributed apis such as `torch.nn.parallel.DistributedDataParallel` initially performs sanity checks that involve ccl ops (allgather) on cpu tensors. Currently, when using DDP with a custom ccl backend that only supports the custom device (and not cpu), user must explicitly provide `cpu:gloo,custom_device:custom_backend`.
Allowing developers to set `cpu:gloo,custom_device:custom_backend` as the default would improve user experience.
Proposed changes:
implementation is quite simple. Reference: https://github.com/pytorch/pytorch/commit/5f20360b035848e2e64e80bda0268bb8a8eb91a2
- possibly semantics change for `get_default_backend_for_device`: from what I found its result is only used as the backend argument to `init_process_group` which works fine.
### Alternatives
_No response_
### Additional context
_No response_
cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim @dcci
Contributor guide
Assessment
This issue has not been assessed yet.