intel / intel/torch-xpu-ops

[release/2.14] UT inductor c10d distributed collective operators not implemented for XPU (DTensor + compiled autograd)

Open
#4,949 0 comments 0 reactions 0 assignees View on GitHub
os: Windows test: ut
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

### 🐛 Describe the bug

Issue found during pre RC test with torch 2.14.0.dev+xpu on Windows with Intel XPU.

20 UTs from `pytorch\test\inductor\test_compiled_autograd.py` (class `TestDTensorCompileWithCompiledAutograd`) fail on Windows with Intel XPU because several `c10d` distributed collective operators are **not implemented for the XPU device**.

5 distinct operators are missing:

```
c10d::allgather_into_tensor_coalesced_ (10 tests)
c10d::scatter_ ( 5 tests)
c10d::broadcast_ ( 2 tests)
c10d::reduce_scatter_tensor_coalesced_ ( 2 tests)
c10d::allreduce_ ( 1 test)
```

### Affected Test Cases

```
# c10d::allgather_into_tensor_coalesced_
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_compile_redistribute_flattened_mesh - NotImplementedError: The operator 'c10d::allgather_into_tensor_coalesced_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_different_gradient_placement - NotImplementedError: The operator 'c10d::allgather_into_tensor_coalesced_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_dynamic - NotImplementedError: The operator 'c10d::allgather_into_tensor_coalesced_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_dynamic_cat - NotImplementedError: The operator 'c10d::allgather_into_tensor_coalesced_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_dynamic_slice - NotImplementedError: The operator 'c10d::allgather_into_tensor_coalesced_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_matmul_cost_hint_or_upper_bound - NotImplementedError: The operator 'c10d::allgather_into_tensor_coalesced_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dynamo_dtensor_from_local_redistribute_async - NotImplementedError: The operator 'c10d::allgather_into_tensor_coalesced_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_graph_input_is_async - NotImplementedError: The operator 'c10d::allgather_into_tensor_coalesced_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_to_local_symbolic_sizes_non_sharded_dims - NotImplementedError: The operator 'c10d::allgather_into_tensor_coalesced_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_to_local_symbolic_sizes_uneven_shard - NotImplementedError: The operator 'c10d::allgather_into_tensor_coalesced_' is not currently implemented for the XPU device.

# c10d::scatter_
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_requires_grad_intermediate_backward - NotImplementedError: The operator 'c10d::scatter_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_requires_grad_recompile - NotImplementedError: The operator 'c10d::scatter_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_make_fx_tp_embedding_no_shadow_nodes - NotImplementedError: The operator 'c10d::scatter_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_tp_compile_comm_reordering - NotImplementedError: The operator 'c10d::scatter_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_tp_compile_comm_reordering_graph_partition - NotImplementedError: The operator 'c10d::scatter_' is not currently implemented for the XPU device.

# c10d::broadcast_
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_matmul_zero_size_shards - NotImplementedError: The operator 'c10d::broadcast_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_unbacked_matmuls - NotImplementedError: The operator 'c10d::broadcast_' is not currently implemented for the XPU device.

# c10d::reduce_scatter_tensor_coalesced_
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent - NotImplementedError: The operator 'c10d::reduce_scatter_tensor_coalesced_' is not currently implemented for the XPU device.
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides - NotImplementedError: The operator 'c10d::reduce_scatter_tensor_coalesced_' is not currently implemented for the XPU device.

# c10d::allreduce_
test_compiled_autograd.py::TestDTensorCompileWithCompiledAutograd::test_dtensor_partial_placement_graph_output - NotImplementedError: The operator 'c10d::allreduce_' is not currently implemented for the XPU device.
```

### Error Message

click to expand

```
______ TestDTensorCompileWithCompiledAutograd.test_dtensor_dynamic_slice ______
Traceback (most recent call last):
File "C:\Users\gta\repositories\pytorch\pytorch\test\distributed\tensor\test_dtensor_compile.py", line 915, in test_dtensor_dynamic_slice
ref = fn(x)
^^^^^
File "C:\Users\gta\repositories\pytorch\pytorch\test\distributed\tensor\test_dtensor_compile.py", line 909, in fn
for t in torch.tensor_split(x, 2)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\gta\miniforge3\envs\pytorch_2.14_pre-release\Lib\site-packages\torch\distributed\tensor\_dispatch.py", line 357, in _dispatch_get_local_results_slow_path
self.redistribute_local_args(
File "C:\Users\gta\miniforge3\envs\pytorch_2.14_pre-release\Lib\site-packages\torch\distributed\tensor\_dispatch.py", line 648, in redistribute_local_args
resharded_local_tensor = redistribute_local_tensor(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\gta\miniforge3\envs\pytorch_2.14_pre-release\Lib\site-packages\torch\distributed\tensor\_redistribute.py", line 1655, in redistribute_local_tensor
new_local_tensor = current_placement._to_replicate_tensor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\gta\miniforge3\envs\pytorch_2.14_pre-release\Lib\site-packages\torch\distributed\tensor\placement_types.py", line 609, in _to_replicate_tensor
result = funcol.all_gather_single(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\gta\miniforge3\envs\pytorch_2.14_pre-release\Lib\site-packages\torch\distributed\_functional_collectives.py", line 209, in all_gather_single
tensor = torch.ops._c10d_functional.all_gather_into_tensor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\gta\miniforge3\envs\pytorch_2.14_pre-release\Lib\site-packages\torch\_ops.py", line 1350, in __call__
return self._op(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
NotImplementedError: The operator 'c10d::allgather_into_tensor_coalesced_' is not currently implemented for the XPU device. Please open a feature on https://github.com/intel/torch-xpu-ops/issues. You can set the environment variable `PYTORCH_ENABLE_XPU_FALLBACK=1` to use the CPU implementation as a fallback for XPU unimplemented operators. WARNING: this will bring unexpected performance compared with running natively on XPU.
```

### Versions

Click to expand traceback
PyTorch version: 2.14.0.dev20260805+xpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 11 Pro (10.0.26200 64-bit)
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.31.6
Libc version: N/A

Python version: 3.12.13 | packaged by conda-forge | (main, Mar 5 2026, 16:36:12) [MSC v.1944 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-11-10.0.26200-SP0
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: True
XPU used to build PyTorch: 20260100
Intel GPU driver version:
* 32.0.101.8864 (20260717000000.******+***)
Intel GPU models onboard:
* Intel(R) Arc(TM) B580 Graphics
Intel GPU models detected:
* [0] _XpuDeviceProperties(name='Intel(R) Arc(TM) B580 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE20B, uuid=86800be2-0000-0000-0300-000000000000, driver_version='1.15.38308', total_memory=11875MB, local_mem_size=128KB, last_level_cache_size=18432KB, max_compute_units=160, memory_clock_rate=0MHz, memory_bus_width=64-bit, gpu_eu_count=160, gpu_subslice_count=20, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False
Caching allocator config: N/A

CPU:
Name: 13th Gen Intel(R) Core(TM) i5-13400
Manufacturer: GenuineIntel
Family: 205
Architecture: 9
ProcessorType: 3
DeviceID: CPU0
CurrentClockSpeed: 2500
MaxClockSpeed: 2500
L2CacheSize: 9728
L2CacheSpeed: None
Revision: None

Versions of relevant libraries:
[pip3] dpcpp-cpp-rt==2026.1.0
[pip3] intel-cmplr-lib-rt==2026.1.0
[pip3] intel-cmplr-lib-ur==2026.1.0
[pip3] intel-cmplr-lic-rt==2026.1.0
[pip3] intel-opencl-rt==2026.1.0
[pip3] intel-openmp==2026.1.0
[pip3] intel-pti==1.0.1
[pip3] intel-sycl-rt==2026.1.0
[pip3] mkl==2026.1.0
[pip3] mkl-include==2024.2.0
[pip3] mkl-static==2024.2.0
[pip3] mypy_extensions==1.1.0
[pip3] numpy==1.26.4
[pip3] onemkl-license==2026.1.0
[pip3] onemkl-sycl-blas==2026.1.0
[pip3] onemkl-sycl-dft==2026.1.0
[pip3] onemkl-sycl-lapack==2026.1.0
[pip3] onemkl-sycl-rng==2026.1.0
[pip3] onemkl-sycl-sparse==2026.1.0
[pip3] onnx==1.21.0
[pip3] onnx-ir==0.1.16
[pip3] onnxscript==0.6.2
[pip3] optree==0.13.0
[pip3] tbb==2023.1.0
[pip3] tbb-devel==2021.13.1
[pip3] tcmlib==1.5.0
[pip3] torch==2.14.0.dev20260805+xpu
[pip3] torchaudio==2.11.0.dev20260805+xpu
[pip3] torchvision==0.29.0.dev20260805+xpu
[pip3] triton-xpu==3.8.0+git1e2d42a0
[pip3] umf==1.1.0
[conda] dpcpp-cpp-rt 2026.1.0 pypi_0 pypi
[conda] intel-cmplr-lib-rt 2026.1.0 pypi_0 pypi
[conda] intel-cmplr-lib-ur 2026.1.0 pypi_0 pypi
[conda] intel-cmplr-lic-rt 2026.1.0 pypi_0 pypi
[conda] intel-opencl-rt 2026.1.0 pypi_0 pypi
[conda] intel-openmp 2026.1.0 pypi_0 pypi
[conda] intel-pti 1.0.1 pypi_0 pypi
[conda] intel-sycl-rt 2026.1.0 pypi_0 pypi
[conda] mkl 2026.1.0 pypi_0 pypi
[conda] mkl-include 2024.2.0 pypi_0 pypi
[conda] mkl-static 2024.2.0 pypi_0 pypi
[conda] numpy 1.26.4 pypi_0 pypi
[conda] onemkl-license 2026.1.0 pypi_0 pypi
[conda] onemkl-sycl-blas 2026.1.0 pypi_0 pypi
[conda] onemkl-sycl-dft 2026.1.0 pypi_0 pypi
[conda] onemkl-sycl-lapack 2026.1.0 pypi_0 pypi
[conda] onemkl-sycl-rng 2026.1.0 pypi_0 pypi
[conda] onemkl-sycl-sparse 2026.1.0 pypi_0 pypi
[conda] optree 0.13.0 pypi_0 pypi
[conda] tbb 2023.1.0 pypi_0 pypi
[conda] tbb-devel 2021.13.1 pypi_0 pypi
[conda] tcmlib 1.5.0 pypi_0 pypi
[conda] torch 2.14.0.dev20260805+xpu pypi_0 pypi
[conda] torchaudio 2.11.0.dev20260805+xpu pypi_0 pypi
[conda] torchvision 0.29.0.dev20260805+xpu pypi_0 pypi
[conda] triton-xpu 3.8.0+git1e2d42a0 pypi_0 pypi
[conda] umf 1.1.0 pypi_0 pypi

Contributor guide

Open the contributing guide

Research direction

Start with the listed TestDTensorCompileWithCompiledAutograd cases in test_compiled_autograd.py and the traceback location in test/distributed/tensor/test_dtensor_compile.py. Run the failures on Windows with Intel XPU, then inspect how the five named c10d collective operators are implemented for XPU. Done means all 20 affected tests pass natively without PYTORCH_ENABLE_XPU_FALLBACK.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.