intel / intel/torch-xpu-ops

[XPU] torchrun --numa-binding fails on XPU: _XpuDeviceProperties missing pci_domain_id/pci_bus_id/pci_device_id

Open
#4,732 0 comments 0 reactions 1 assignee Claimed by @astachowiczhabana View on GitHub
agent:active test: ut
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

### 🐛 Describe the bug

### 🐛 Describe the bug

`torch/numa/binding.py` was made device-generic in #185266 (`torch.cuda` ->
`torch.accelerator`), but XPU doesn't satisfy the implicit backend contract, so
`--numa-binding` is unusable on Intel GPUs.

`_get_numa_node_index_for_device_index` (`torch/numa/binding.py:595-604`) reads
three PCI attributes to find the device's NUMA node via sysfs:

```python
domain = device_properties.pci_domain_id
bus = device_properties.pci_bus_id
device = device_properties.pci_device_id
pci_addr = f"{domain:04x}:{bus:02x}:{device:02x}.0"
with open(f"/sys/bus/pci/devices/{pci_addr}/numa_node") as f: ...
```

CUDA binds them at `torch/csrc/cuda/Module.cpp:1201-1203`. XPU exposes no `pci_*`
attribute at all (full list: `torch/csrc/xpu/Module.cpp:326-360`), and
`grep -rn "pci" torch/csrc/xpu/ torch/xpu/ c10/xpu/` returns zero hits. The data is
missing one layer down too — `AT_FORALL_XPU_EXT_DEVICE_PROPERTIES`
(`c10/xpu/XPUDeviceProp.h:116-139`) never queries a PCI address — so *
need a change**, not just the pybind11 binding.

### Impact

`torchrun --numa-binding=` **fails to launch** with `AttributeError`:
`torch/distributed/run.py:957-960` doesn't set
`should_fall_back_if_binding_fails`, which defaults to `False`
(`binding.py:95`), so `_handle_exception` re-raises (`binding.py:266`)
fallback enabled it instead runs unbound and loses the locality win. The upstream
default (`numa_options=None`) is unaffected.

### Secondary issue: no test enforces the contract

`test/test_numa_binding.py` passes on XPU but gives no real XPU covera
`torch.get_device_module` to a `MagicMock` returning a hand-rolled
`MockDeviceProperties` that declares the three `pci_*` fields (lines 2
It would pass with no accelerator present. That mocking is right for its purpose
(fabricating 4-socket / 8-NUMA-node topologies no CI runner has), so t
missing test at a different layer. The contract exists only in #185266's
description:

> backends ... must expose `get_device_properties()` returning an obje
> `pci_domain_id`, `pci_bus_id`, and `pci_device_id`

`test/test_xpu.py:131` runs on real hardware but only checks self-consistency and a
few `> 0` assertions, so it misses this too.

### Suggested direction

1. Expose the PCI address on XPU. SYCL appears to offer it as one stri
(`"0000:4d:00.0"`) not three ints — confirm against the installed
`sycl/ext/intel/info/device.hpp`, including drivers lacking the ext
Parsing to ints in the XPU layer keeps `binding.py` single-shaped.
2. Add a real-hardware contract test asserting those attributes exist
`torch.accelerator.current_accelerator()`'s properties.
`test/test_accelerator.py:277` is a good template.

HPU and PrivateUse1/OpenReg are likely affected the same way.

### Versions

`main` at f594cadbd10e.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.