[XPU] torchrun --numa-binding fails on XPU: _XpuDeviceProperties missing pci_domain_id/pci_bus_id/pci_device_id
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
### 🐛 Describe the bug
### 🐛 Describe the bug
`torch/numa/binding.py` was made device-generic in #185266 (`torch.cuda` ->
`torch.accelerator`), but XPU doesn't satisfy the implicit backend contract, so
`--numa-binding` is unusable on Intel GPUs.
`_get_numa_node_index_for_device_index` (`torch/numa/binding.py:595-604`) reads
three PCI attributes to find the device's NUMA node via sysfs:
```python
domain = device_properties.pci_domain_id
bus = device_properties.pci_bus_id
device = device_properties.pci_device_id
pci_addr = f"{domain:04x}:{bus:02x}:{device:02x}.0"
with open(f"/sys/bus/pci/devices/{pci_addr}/numa_node") as f: ...
```
CUDA binds them at `torch/csrc/cuda/Module.cpp:1201-1203`. XPU exposes no `pci_*`
attribute at all (full list: `torch/csrc/xpu/Module.cpp:326-360`), and
`grep -rn "pci" torch/csrc/xpu/ torch/xpu/ c10/xpu/` returns zero hits. The data is
missing one layer down too — `AT_FORALL_XPU_EXT_DEVICE_PROPERTIES`
(`c10/xpu/XPUDeviceProp.h:116-139`) never queries a PCI address — so *
need a change**, not just the pybind11 binding.
### Impact
`torchrun --numa-binding=` **fails to launch** with `AttributeError`:
`torch/distributed/run.py:957-960` doesn't set
`should_fall_back_if_binding_fails`, which defaults to `False`
(`binding.py:95`), so `_handle_exception` re-raises (`binding.py:266`)
fallback enabled it instead runs unbound and loses the locality win. The upstream
default (`numa_options=None`) is unaffected.
### Secondary issue: no test enforces the contract
`test/test_numa_binding.py` passes on XPU but gives no real XPU covera
`torch.get_device_module` to a `MagicMock` returning a hand-rolled
`MockDeviceProperties` that declares the three `pci_*` fields (lines 2
It would pass with no accelerator present. That mocking is right for its purpose
(fabricating 4-socket / 8-NUMA-node topologies no CI runner has), so t
missing test at a different layer. The contract exists only in #185266's
description:
> backends ... must expose `get_device_properties()` returning an obje
> `pci_domain_id`, `pci_bus_id`, and `pci_device_id`
`test/test_xpu.py:131` runs on real hardware but only checks self-consistency and a
few `> 0` assertions, so it misses this too.
### Suggested direction
1. Expose the PCI address on XPU. SYCL appears to offer it as one stri
(`"0000:4d:00.0"`) not three ints — confirm against the installed
`sycl/ext/intel/info/device.hpp`, including drivers lacking the ext
Parsing to ints in the XPU layer keeps `binding.py` single-shaped.
2. Add a real-hardware contract test asserting those attributes exist
`torch.accelerator.current_accelerator()`'s properties.
`test/test_accelerator.py:277` is a good template.
HPU and PrivateUse1/OpenReg are likely affected the same way.
### Versions
`main` at f594cadbd10e.
Contributor guide
Assessment
This issue has not been assessed yet.