[distributed] No XPU equivalent of NCCL Copy Engine collectives, so test_ce_colls.py cannot run on XPU
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
### 🐛 Describe the bug
`test/distributed/test_ce_colls.py` exercises NCCL's *copy-engine* (CE) collectives. It
is unreachable on an XPU build: the module exits at import, before any test is collected.
```python
if not dist.is_available() or not dist.is_nccl_available():
print("c10d NCCL not available, skipping tests", file=sys.stderr)
sys.exit(0)
```
Current XPU state: exits with `c10d NCCL not available, skipping tests` (0 tests
collected).
Beyond that guard the tests are NCCL-specific by construction:
- `@requires_nccl_version((2, 28), "Need NCCL 2.28+ for CE collectives")` on the class
- `@requires_cuda_p2p_access()` on the class
- `symm_mem.set_backend("NCCL")` in setup, i.e. they select the NCCL symmetric-memory
backend explicitly rather than using the generic symmetric-memory API
So this is not a test-side port: it would need CE-equivalent functionality in XCCL (and a
way to select it through `symm_mem.set_backend`). Filing it to record the gap rather than
to claim the test is nearly portable.
For contrast, the generic symmetric-memory path *is* available on XPU --
`symm_mem.get_backend(torch.device("xpu"))` returns `"XPU"` (implemented in
`src/xccl/XPUSymmetricMemory.cpp`) -- so `test/distributed/test_symmetric_memory.py` can
be ported while this file cannot.
Found while auditing `test/distributed/` for device-agnostic XPU support
(pytorch/pytorch#114850).
### What must pass to close this
`test/distributed/test_ce_colls.py` -- the module must get past its import-time
`is_nccl_available()` exit and both tests must run and pass on XPU:
```bash
python test/distributed/test_ce_colls.py -v
```
- `NCCLCopyEngineCollectives::test_ce_allgather`
- `NCCLCopyEngineCollectives::test_ce_alltoall`
Today the file collects 0 tests on XPU. Note the class name itself encodes the NCCL
dependency, so closing this realistically means either an XCCL CE-equivalent selectable
through `symm_mem.set_backend`, or agreement that the file stays CUDA-only and the issue
is closed as won't-fix -- worth deciding before any work starts.
### Versions
```
torch 2.15.0a0+git3c3da2a (source build, commit 3c3da2a5e8b)
device 4x Intel(R) Data Center GPU Max 1100
backend xccl
```
Contributor guide
Research direction
Start with test/distributed/test_ce_colls.py and its import guard, NCCL version and P2P requirements, then read src/xccl/XPUSymmetricMemory.cpp and the XCCL symmetric-memory backend selection. First decide whether XCCL will provide CE-equivalent collectives selectable through symm_mem.set_backend or whether the issue is won't-fix. Done means the stated test command runs both named tests successfully on XPU, or the CUDA-only decision is recorded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- distributed-systems, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100