pytorch / pytorch/pytorch

add support for distributed backends to specify default backend in multiple backends format

Open
#164,925 2 comments 0 reactions 1 assignee Claimed by @kwen2501 View on GitHub
enhancement oncall: distributed triaged
Dominant language
Python
Stars
103k
Forks
29.5k
PR merge metrics
PR metrics pending

Description

### 🚀 The feature, motivation and pitch

Currently a dictionary exists for in-tree and custom device types to specify default backend to be used with `init_process_group(backend=None)`.

This is defined in `torch.distributed.distributed_c10d.py::Backend.default_device_backend_map`.

It would be useful for developers to specify a default backend in the colon format like: `cpu:gloo,custom_device:custom_backend` for the following reason:

Distributed apis such as `torch.nn.parallel.DistributedDataParallel` initially performs sanity checks that involve ccl ops (allgather) on cpu tensors. Currently, when using DDP with a custom ccl backend that only supports the custom device (and not cpu), user must explicitly provide `cpu:gloo,custom_device:custom_backend`.

Allowing developers to set `cpu:gloo,custom_device:custom_backend` as the default would improve user experience.

Proposed changes:
implementation is quite simple. Reference: https://github.com/pytorch/pytorch/commit/5f20360b035848e2e64e80bda0268bb8a8eb91a2

- possibly semantics change for `get_default_backend_for_device`: from what I found its result is only used as the backend argument to `init_process_group` which works fine.

### Alternatives

_No response_

### Additional context

_No response_

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim @dcci

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.