[distributed] ProcessGroupXCCL does not implement supportsTimeEstimation(), so collective time estimation is unavailable on XPU
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
### 🐛 Describe the bug
`ProcessGroupXCCL` does not override the experimental collective time-estimation API,
so it inherits `return false` from `Backend.hpp`. As a result
`torch.distributed._time_estimator()` raises on XPU, and inductor's NCCL-based
collective estimator silently declines to run.
```bash
python - <<'EOF'
import os, torch, torch.distributed as dist
os.environ.update(MASTER_ADDR="localhost", MASTER_PORT="29711")
dist.init_process_group("xccl", rank=0, world_size=1, store=dist.HashStore())
b = dist.group.WORLD._get_backend(torch.device("xpu"))
print("backend:", type(b).__name__)
print("_supports_time_estimate:", b._supports_time_estimate)
dist.destroy_process_group()
EOF
```
```
backend: ProcessGroupXCCL
_supports_time_estimate: False
```
The capability is a virtual with a `false` default in
`torch/csrc/distributed/c10d/Backend.hpp:175`:
```cpp
// Experimental collective time-estimation API. Backends that return true
// must simulate collectives issued between startTimeEstimate() and
// endTimeEstimate(), which returns the estimated duration in microseconds.
virtual bool supportsTimeEstimation() const {
return false;
}
virtual void startTimeEstimate() {
TORCH_CHECK(false, c10::str("Backend ", getBackendName(),
" does not support time estimation"));
}
virtual float endTimeEstimate() {
TORCH_CHECK(false, c10::str("Backend ", getBackendName(),
" does not support time estimation"));
}
```
Only `ProcessGroupNCCL.hpp:839` overrides it. `grep -rn supportsTimeEstimation
src/xccl/` in this repo returns nothing.
### Impact
Two consumers in PyTorch:
1. `torch/distributed/distributed_c10d.py:3773` — `dist._time_estimator()` raises
`NotImplementedError: collective time estimator is not supported in the current
version of backend ...`, so the public estimator context manager is unusable on XPU.
2. `torch/_inductor/comm_analysis.py:868` — `if not backend._supports_time_estimate:
return None`, so inductor falls back to its analytical formula instead of a measured
estimate when scheduling/reordering collectives on XPU.
### What XCCL needs
`supportsTimeEstimation()` returning `true`, plus `startTimeEstimate()` /
`endTimeEstimate()` (the base class `TORCH_CHECK`s on both), simulating the collectives
issued between them and returning the estimated duration in microseconds.
### Note: a second, PyTorch-side blocker
Even once XCCL implements the above, inductor's estimator will not reach it —
`torch/_inductor/comm_analysis.py:863` hardcodes the device before looking up the
backend:
```python
device = torch.device("cuda")
try:
backend = pg._get_backend(device)
except RuntimeError:
return None
```
On an XPU-only build `pg._get_backend(torch.device("cuda"))` raises `RuntimeError` and
the function returns `None`. That line needs to follow the current accelerator; it is a
PyTorch fix rather than a torch-xpu-ops one, and is mentioned here only so the two are
not fixed in isolation.
### Test impact
In `test/inductor/test_comm_analysis.py`, two tests assert this capability:
- `TestNcclEstimateDeviceResolution::test_multi_backend_pg_resolves_to_nccl`
- `TestNcclEstimateDeviceResolution::test_single_nccl_backend_resolves_correctly`
Both do `self.assertTrue(backend._supports_time_estimate)`, so they cannot pass on XPU
today; while porting that file for device-agnosticism (pytorch/pytorch#114850) they are
being marked `@skipIfXpu` referencing this issue. The file's third test,
`test_fake_backend_falls_back_to_analytical`, is backend-agnostic and now runs on XPU
(it was previously excluded by a module-level `if not dist.is_nccl_available():
sys.exit(0)` guard).
### Versions
```
torch 2.15.0a0+git3c3da2a (source build, commit 3c3da2a5e8b)
device 4x Intel(R) Data Center GPU Max 1100
backend xccl
```
Contributor guide
Research direction
Start by locating ProcessGroupXCCL under src/xccl and compare its backend API with ProcessGroupNCCL.hpp:839 and the defaults in torch/csrc/distributed/c10d/Backend.hpp. Trace torch/distributed/distributed_c10d.py:3773 and torch/_inductor/comm_analysis.py:863-868, then inspect test/inductor/test_comm_analysis.py. Done means XCCL reports and performs time estimation, while the separate current-accelerator blocker is identified for its PyTorch-side fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend-api-design, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100