intel / intel/torch-xpu-ops

[distributed] ProcessGroupXCCL does not implement supportsTimeEstimation(), so collective time estimation is unavailable on XPU

Open
#5,405 0 comments 0 reactions 0 assignees View on GitHub
bug module: distributed
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

### 🐛 Describe the bug

`ProcessGroupXCCL` does not override the experimental collective time-estimation API,
so it inherits `return false` from `Backend.hpp`. As a result
`torch.distributed._time_estimator()` raises on XPU, and inductor's NCCL-based
collective estimator silently declines to run.

```bash
python - <<'EOF'
import os, torch, torch.distributed as dist
os.environ.update(MASTER_ADDR="localhost", MASTER_PORT="29711")
dist.init_process_group("xccl", rank=0, world_size=1, store=dist.HashStore())
b = dist.group.WORLD._get_backend(torch.device("xpu"))
print("backend:", type(b).__name__)
print("_supports_time_estimate:", b._supports_time_estimate)
dist.destroy_process_group()
EOF
```

```
backend: ProcessGroupXCCL
_supports_time_estimate: False
```

The capability is a virtual with a `false` default in
`torch/csrc/distributed/c10d/Backend.hpp:175`:

```cpp
// Experimental collective time-estimation API. Backends that return true
// must simulate collectives issued between startTimeEstimate() and
// endTimeEstimate(), which returns the estimated duration in microseconds.
virtual bool supportsTimeEstimation() const {
return false;
}

virtual void startTimeEstimate() {
TORCH_CHECK(false, c10::str("Backend ", getBackendName(),
" does not support time estimation"));
}

virtual float endTimeEstimate() {
TORCH_CHECK(false, c10::str("Backend ", getBackendName(),
" does not support time estimation"));
}
```

Only `ProcessGroupNCCL.hpp:839` overrides it. `grep -rn supportsTimeEstimation
src/xccl/` in this repo returns nothing.

### Impact

Two consumers in PyTorch:

1. `torch/distributed/distributed_c10d.py:3773` — `dist._time_estimator()` raises
`NotImplementedError: collective time estimator is not supported in the current
version of backend ...`, so the public estimator context manager is unusable on XPU.
2. `torch/_inductor/comm_analysis.py:868` — `if not backend._supports_time_estimate:
return None`, so inductor falls back to its analytical formula instead of a measured
estimate when scheduling/reordering collectives on XPU.

### What XCCL needs

`supportsTimeEstimation()` returning `true`, plus `startTimeEstimate()` /
`endTimeEstimate()` (the base class `TORCH_CHECK`s on both), simulating the collectives
issued between them and returning the estimated duration in microseconds.

### Note: a second, PyTorch-side blocker

Even once XCCL implements the above, inductor's estimator will not reach it —
`torch/_inductor/comm_analysis.py:863` hardcodes the device before looking up the
backend:

```python
device = torch.device("cuda")
try:
backend = pg._get_backend(device)
except RuntimeError:
return None
```

On an XPU-only build `pg._get_backend(torch.device("cuda"))` raises `RuntimeError` and
the function returns `None`. That line needs to follow the current accelerator; it is a
PyTorch fix rather than a torch-xpu-ops one, and is mentioned here only so the two are
not fixed in isolation.

### Test impact

In `test/inductor/test_comm_analysis.py`, two tests assert this capability:

- `TestNcclEstimateDeviceResolution::test_multi_backend_pg_resolves_to_nccl`
- `TestNcclEstimateDeviceResolution::test_single_nccl_backend_resolves_correctly`

Both do `self.assertTrue(backend._supports_time_estimate)`, so they cannot pass on XPU
today; while porting that file for device-agnosticism (pytorch/pytorch#114850) they are
being marked `@skipIfXpu` referencing this issue. The file's third test,
`test_fake_backend_falls_back_to_analytical`, is backend-agnostic and now runs on XPU
(it was previously excluded by a module-level `if not dist.is_nccl_available():
sys.exit(0)` guard).

### Versions

```
torch 2.15.0a0+git3c3da2a (source build, commit 3c3da2a5e8b)
device 4x Intel(R) Data Center GPU Max 1100
backend xccl
```

Contributor guide

Open the contributing guide

Research direction

Start by locating ProcessGroupXCCL under src/xccl and compare its backend API with ProcessGroupNCCL.hpp:839 and the defaults in torch/csrc/distributed/c10d/Backend.hpp. Trace torch/distributed/distributed_c10d.py:3773 and torch/_inductor/comm_analysis.py:863-868, then inspect test/inductor/test_comm_analysis.py. Done means XCCL reports and performs time estimation, while the separate current-accelerator blocker is identified for its PyTorch-side fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.