intel / intel/torch-xpu-ops

[xccl] ProcessGroupXCCL inherits unimplemented Backend virtuals (splitting, shrinking, suspend/resume, memory stats, comm_split_count)

Open
#5,385 0 comments 0 reactions 1 assignee Claimed by @frost-intel View on GitHub
module: distributed
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

> Drafted with assistance from an AI coding agent (Claude Code) and reviewed/verified by me on 4x Intel Data Center GPU Max 1550.

### Summary

`ProcessGroupXCCL` does not override a family of `Backend` virtuals, so each one
silently inherits a `false` or throwing default. The Python bindings remain
visible on `ProcessGroupXCCL`, so the capabilities look supported until they are
used.

This is an umbrella for the pattern; #5382 and #5383 are two already-filed
instances of it.

### Unimplemented virtuals

| Virtual | Default in `torch/csrc/distributed/c10d/Backend.hpp` | Effect |
|---|---|---|
| `supportsSplitting` | `:157` returns `false` | `split_group` fails: "No backend for the parent process group or its backend does not support splitting" |
| `supportsTimeEstimation` | `:175` returns `false` | see #5383 |
| `supportsShrinking` | `:193` returns `false` | `shrink_group` raises `TypeError: Process group backend ... does not support shrinking operations` |
| `registerOnCompletionHook` | `:672` `TORCH_CHECK(false)` | see #5382 |
| `waitForPendingWorks` | `:681` `TORCH_CHECK(false)` | untested, same shape |
| `suspend` | `:795` `TORCH_CHECK(false)` | "Backend xccl does not support suspend" |
| `resume` | `:801` `TORCH_CHECK(false)` | as above |
| `getMemoryStats` | `:807` `TORCH_CHECK(false)` | "Backend xccl does not support getMemoryStats" |

Two adjacent gaps of a slightly different kind:

- **`comm_split_count`** is bound for `ProcessGroupNCCL` only
(`torch/csrc/distributed/c10d/init.cpp:3937`), so
`AttributeError: 'ProcessGroupXCCL' object has no attribute 'comm_split_count'`.
- **`Work::getTimeout`** (`torch/csrc/distributed/c10d/Work.cpp:197`) throws
"This Backend doesn't support getTimeout." This is on `Work`, not `Backend`.

### Reproduction

PyTorch `2.15.0a0+git4d3ac9e`, oneCCL 2022.1, 4x Intel Data Center GPU Max 1550.

### Impact

24 tests in `ProcessGroupNCCLGroupTest` (`test/distributed/test_c10d_nccl.py`)
cannot be enabled for XPU:

- splitting (6): `test_comm_split_group`, `..._backend_filter`,
`..._backend_validation`, `..._mixed_backend`, `..._out_of_order_ranks`,
`test_comm_split_initialized_parent_with_lazy_default`
- shrinking (8): `test_shrink_group_basic`, `..._flags`, `..._backend_properties`,
`..._multiple_comms`, `..._multiple_exclusions`, `..._multiple_iterations`,
`..._performance`, `..._vs_abort_reinit_performance`
- `comm_split_count` (5): `test_comm_split_subgroup`, `test_non_blocking_init`,
`test_non_blocking_with_eager_init`, `test_abort_in_destroy_multi_pgs`,
`test_abort_in_destroy_mixed_empty_pgs`
- suspend/resume (2): `test_suspend`, `test_resume`
- memory stats (1): `test_get_memory_stats`
- `Work::getTimeout` (2): `test_extend_nccl_pg_timeout_backend0`,
`test_extend_nccl_pg_timeout_backend_xccl`

### Missing TORCH_XCCL_* environment knobs

NCCL exposes several policy knobs that have no `TORCH_XCCL_*` counterpart, so the
corresponding behaviour cannot be exercised or configured on XPU:

| NCCL env var | XCCL equivalent | Note |
|---|---|---|
| `TORCH_NCCL_PROPAGATE_ERROR` | none | error-propagation policy |
| `TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC` | none | watchdog heartbeat / dump-on-timeout |
| `TORCH_NCCL_ASYNC_ERROR_HANDLING` | none | async error handling mode |

(`TORCH_NCCL_TRACE_BUFFER_SIZE` and `TORCH_NCCL_DEBUG_INFO_TEMP_FILE` do have
backend-neutral `TORCH_FR_*` spellings and are fine.)

Blocks 9 further tests, ported to be device-agnostic but gated with `@skipIfXpu`:

- `NcclErrorHandlingTest` (8): error-propagation and blocking-wait behaviour
- `NcclErrorDumpTest` (1): dump-on-timeout via the heartbeat monitor

Also NCCL-library-specific and not portable: `NcclUserBufferRegistrationTest` (2)
needs `NCCL_ALGO`/`NCCL_DEBUG`/`NCCL_DEBUG_SUBSYS` and scrapes NCCL debug logs;
`ProcessGroupNCCLLargerScaleTest` (3) needs `comm_split_count`.

### Further gaps outside the Backend-virtual family

These five are distinct root causes, each blocking tests in
`ProcessGroupNCCLGroupTest` that are left failing (not skipped) so the gap stays
visible in CI:

| Gap | Detail | Tests |
|---|---|---|
| `_get_process_group_uid` is NCCL-only | `torch/distributed/distributed_c10d.py:2224` returns `-1` unless `isinstance(backend, ProcessGroupNCCL)`, even though `ProcessGroupXCCL.uid` exists. Already noted by @guangyey on pytorch/pytorch#183625: "This function name is `_get_process_group_uid` but only return backend `ProcessGroupNCCL`'s uid. It doesn't make sense." | `test_get_uid` |
| no `_DEFAULT_PG_XCCL_TIMEOUT` | `torch.distributed.constants.default_pg_nccl_timeout` falls back to `None` when `_DEFAULT_PG_NCCL_TIMEOUT` is absent, so tests comparing against it see `None` while the actual default is `0:30:00` | `test_init_process_group_nccl_timeout` |
| `c10d::gather_into_tensor_` unregistered for XPU | `NotImplementedError: The operator 'c10d::gather_into_tensor_' is not currently implemented for the XPU device` | `test_gather_single` |
| XCCL does not raise `DistBackendError` | abort/error paths surface a different exception type than NCCL | `test_abort_in_destroy_pg`, `test_nccl_dist_backend_error` |
| extra-context check is NVML-specific | `_helper_test_extra_cuda_context_by_nvml` uses `pynvml`; the memory-based fallback also measures CUDA context. Likely not portable as written | `test_extra_cuda_context` |

Note `c10d::allreduce_` also has no `SparseXPU` registration, which blocks
`SparseCollective::test_ddp_set_sparse_metadata` and
`ProcessGroupNCCLOpTest::test_sparse_allreduce_ops` (both skipped with
`@skipIfXpu`).

**Before closing this ticket, remove the corresponding `@skipIfXpu` lines from the upstream tests.**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.