intel / intel/llvm

[v7.0.0] AssertionError: Tensor-likes are not equal on Pytorch built with dpclang v7.0.0

Open
#22,895 9 comments 0 reactions 0 assignees View on GitHub
bug confirmed
Dominant language
LLVM
Stars
1.5k
Forks
854
Avg merge
3d 17h
Merged PRs (30d)
137

Description

### Describe the bug

Some pytorch tests are failing with on comparing output tensors with the references:
```
AssertionError: Tensor-likes are not equal!
```

### To reproduce

Below assumes that dpclang is installed and configured. For example:
```
export CMAKE_PREFIX_PATH=~/git/intel-llvm/build/install/
export PKG_CONFIG_PATH=$HOME/git/intel-llvm/build/install/lib/pkgconfig:$PKG_CONFIG_PATH
export LD_LIBRARY_PATH=$HOME/git/intel-llvm/build/install/lib:$LD_LIBRARY_PATH
export PATH=$HOME/git/intel-llvm/build/install/bin:$PATH
```

Using pre-built pytorch wheels:

1. Work in python 3.10 virtual environment:
```
uv venv --python 3.10
source .venv/bin/activate
```
2. Install pytorch pre-built with dpclang v7.0.0 by [CI job](https://github.com/intel/torch-xpu-ops/actions/runs/31020166365?pr=4789) (download and install [Torch-XPU-Wheel-4789-31020166365-1-dpclang](https://github.com/intel/torch-xpu-ops/actions/runs/31020166365/artifacts/8938180170) archive)
```
uv pip install torch-2.14.0a0+git99731ed-cp310-cp310-linux_x86_64.whl
```

3. Run the following python script:
```
import torch

input = torch.tensor([
[ 0, 0, 233, 0, 0],
[ 0, 125, 0, 233, 81],
[ 0, 0, 153, 13, 0],
[ 14, 0, 65, 0, 0],
[ 0, 0, 89, 31, 0]], device='xpu:0', dtype=torch.uint8)
src = torch.tensor([
[120, 185, 0, 151, 0],
[143, 106, 24, 6, 0],
[109, 19, 245, 117, 0],
[ 0, 123, 0, 0, 203],
[ 0, 0, 0, 118, 0]], device='xpu:0', dtype=torch.uint8)

index = torch.tensor(
[3, 2, 1, 4, 0], device='xpu:0')

# NOTE: This will actually work correctly:
# print(input.index_add(-1, index, src, alpha=1))

input_bool = input.view(torch.bool)
src_bool = src.view(torch.bool)
print(input_bool.index_add(-1, index, src_bool, alpha=1))
```

With dpclang v7.0.0 output will be:
```
tensor([[False, False, True, False, True],
[False, True, False, True, True],
[False, True, True, True, True],
[ True, False, True, False, False],
[False, False, True, True, False]], device='xpu:0')
```

Expected output is (with pytorch built with oneAPI 2026.0):
```
tensor([[False, False, True, True, True],
[False, True, True, True, True],
[False, True, True, True, True],
[ True, False, True, False, False],
[False, False, True, True, True]], device='xpu:0')
```

To build pytorch from sources:
```
git clone https://github.com/pytorch/pytorch.git && cd pytorch
git reset --hard 240f31ace12a3bbbb34d10666780e1c365cfbb6e
git submodule update --init --recursive
pip install -r requirements.txt
XPU_SYCL_COMPILER=dpclang TORCH_XPU_ARCH_LIST=pvc USE_KINETO=0 USE_ONEMKL_XPU=0 python -m pip install --no-build-isolation -v -e .
```

### Environment

- OS: Linux
- Target device and vendor: Pytorch CI was running on PVC, other devices not tried
- DPC++ version: https://github.com/intel/llvm/releases/tag/v7.0.0
- Dependencies version:
```
$ sycl-ls --verbose
[INFO]: failed to load adapter 'libur_adapter_cuda.so.0' with error: libur_adapter_cuda.so.0: cannot open shared object file: No such file or directory
[INFO]: failed to load adapter '/home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_cuda.so.0' with error: /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_cuda.so.0: cannot open shared object file: No such file or directory
[INFO]: failed to load adapter 'libur_adapter_hip.so.0' with error: libur_adapter_hip.so.0: cannot open shared object file: No such file or directory
[INFO]: failed to load adapter '/home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_hip.so.0' with error: /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_hip.so.0: cannot open shared object file: No such file or directory
[INFO]: loaded adapter 0x0x608a6aae5210 (libur_adapter_level_zero.so.0) from /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_level_zero.so.0
[INFO]: loaded adapter 0x0x608a6aae82f0 (libur_adapter_level_zero_v2.so.0) from /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_level_zero_v2.so.0
[INFO]: failed to load adapter 'libur_adapter_native_cpu.so.0' with error: libur_adapter_native_cpu.so.0: cannot open shared object file: No such file or directory
[INFO]: failed to load adapter '/home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_native_cpu.so.0' with error: /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_native_cpu.so.0: cannot open shared object file: No such file or directory
[INFO]: failed to load adapter 'libur_adapter_offload.so.0' with error: libur_adapter_offload.so.0: cannot open shared object file: No such file or directory
[INFO]: failed to load adapter '/home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_offload.so.0' with error: /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_offload.so.0: cannot open shared object file: No such file or directory
[INFO]: loaded adapter 0x0x608a6aaeb6c0 (libur_adapter_opencl.so.0) from /home/dvrogozh/git/intel-llvm/build/install/lib/libur_adapter_opencl.so.0
[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
[level_zero:gpu][level_zero:1] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
[level_zero:gpu][level_zero:2] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
[level_zero:gpu][level_zero:3] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
[opencl:gpu][opencl:0] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO [25.18.33578]
[opencl:gpu][opencl:1] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO [25.18.33578]
[opencl:gpu][opencl:2] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO [25.18.33578]
[opencl:gpu][opencl:3] Intel(R) OpenCL Graphics, Intel(R) Data Center GPU Max 1550 OpenCL 3.0 NEO [25.18.33578]

Platforms: 2
Platform [#1]:
Version : 1.6
Name : Intel(R) oneAPI Unified Runtime over Level-Zero
Vendor : Intel(R) Corporation
Devices : 4
Device [#0]:
Type : gpu
Version : 12.60.7
Name : Intel(R) Data Center GPU Max 1550
Vendor : Intel(R) Corporation
Driver : 1.6.33578+42
UUID : 8680d50b-2f00-0000-4d00-000000000001
DeviceID : 3029
Num SubDevices : 0
Num SubSubDevices : 0
Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_native_assert ext_intel_gpu_hw_threads_per_eu ext_intel_free_memory ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_external_memory_import ext_oneapi_external_semaphore_import ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_bindless_sampled_image_fetch_1d_usm ext_oneapi_bindless_sampled_image_fetch_1d ext_oneapi_bindless_sampled_image_fetch_2d_usm ext_oneapi_bindless_sampled_image_fetch_2d ext_oneapi_bindless_sampled_image_fetch_3d ext_oneapi_queue_profiling_tag ext_oneapi_virtual_mem ext_oneapi_image_array ext_oneapi_virtual_functions ext_intel_spill_memory_size ext_intel_current_clock_throttle_reasons ext_intel_power_limits ext_oneapi_async_memory_alloc ext_oneapi_exportable_device_mem ext_oneapi_ipc_memory
info::device::sub_group_sizes: 16 32
Architecture: intel_gpu_pvc
Device [#1]:
Type : gpu
Version : 12.60.7
Name : Intel(R) Data Center GPU Max 1550
Vendor : Intel(R) Corporation
Driver : 1.6.33578+42
UUID : 8680d50b-2f00-0000-4d00-000000000002
DeviceID : 3029
Num SubDevices : 0
Num SubSubDevices : 0
Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_native_assert ext_intel_gpu_hw_threads_per_eu ext_intel_free_memory ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_external_memory_import ext_oneapi_external_semaphore_import ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_bindless_sampled_image_fetch_1d_usm ext_oneapi_bindless_sampled_image_fetch_1d ext_oneapi_bindless_sampled_image_fetch_2d_usm ext_oneapi_bindless_sampled_image_fetch_2d ext_oneapi_bindless_sampled_image_fetch_3d ext_oneapi_queue_profiling_tag ext_oneapi_virtual_mem ext_oneapi_image_array ext_oneapi_virtual_functions ext_intel_spill_memory_size ext_intel_current_clock_throttle_reasons ext_intel_power_limits ext_oneapi_async_memory_alloc ext_oneapi_exportable_device_mem ext_oneapi_ipc_memory
info::device::sub_group_sizes: 16 32
Architecture: intel_gpu_pvc
Device [#2]:
Type : gpu
Version : 12.60.7
Name : Intel(R) Data Center GPU Max 1550
Vendor : Intel(R) Corporation
Driver : 1.6.33578+42
UUID : 8680d50b-2f00-0000-9a00-000000000001
DeviceID : 3029
Num SubDevices : 0
Num SubSubDevices : 0
Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_native_assert ext_intel_gpu_hw_threads_per_eu ext_intel_free_memory ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_external_memory_import ext_oneapi_external_semaphore_import ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_bindless_sampled_image_fetch_1d_usm ext_oneapi_bindless_sampled_image_fetch_1d ext_oneapi_bindless_sampled_image_fetch_2d_usm ext_oneapi_bindless_sampled_image_fetch_2d ext_oneapi_bindless_sampled_image_fetch_3d ext_oneapi_queue_profiling_tag ext_oneapi_virtual_mem ext_oneapi_image_array ext_oneapi_virtual_functions ext_intel_spill_memory_size ext_intel_current_clock_throttle_reasons ext_intel_power_limits ext_oneapi_async_memory_alloc ext_oneapi_exportable_device_mem ext_oneapi_ipc_memory
info::device::sub_group_sizes: 16 32
Architecture: intel_gpu_pvc
Device [#3]:
Type : gpu
Version : 12.60.7
Name : Intel(R) Data Center GPU Max 1550
Vendor : Intel(R) Corporation
Driver : 1.6.33578+42
UUID : 8680d50b-2f00-0000-9a00-000000000002
DeviceID : 3029
Num SubDevices : 0
Num SubSubDevices : 0
Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_native_assert ext_intel_gpu_hw_threads_per_eu ext_intel_free_memory ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_oneapi_external_memory_import ext_oneapi_external_semaphore_import ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_bindless_sampled_image_fetch_1d_usm ext_oneapi_bindless_sampled_image_fetch_1d ext_oneapi_bindless_sampled_image_fetch_2d_usm ext_oneapi_bindless_sampled_image_fetch_2d ext_oneapi_bindless_sampled_image_fetch_3d ext_oneapi_queue_profiling_tag ext_oneapi_virtual_mem ext_oneapi_image_array ext_oneapi_virtual_functions ext_intel_spill_memory_size ext_intel_current_clock_throttle_reasons ext_intel_power_limits ext_oneapi_async_memory_alloc ext_oneapi_exportable_device_mem ext_oneapi_ipc_memory
info::device::sub_group_sizes: 16 32
Architecture: intel_gpu_pvc
Platform [#2]:
Version : OpenCL 3.0
Name : Intel(R) OpenCL Graphics
Vendor : Intel(R) Corporation
Devices : 4
Device [#0]:
Type : gpu
Version : OpenCL 3.0 NEO
Name : Intel(R) Data Center GPU Max 1550
Vendor : Intel(R) Corporation
Driver : 25.18.33578
UUID : 8680d50b-2f00-0000-4d00-000000000001
DeviceID : 3029
Num SubDevices : 0
Num SubSubDevices : 0
Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_srgb ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_private_alloca ext_oneapi_atomic16 ext_oneapi_virtual_functions ext_intel_device_info_node_mask
info::device::sub_group_sizes: 16 32
Architecture: intel_gpu_pvc
Device [#1]:
Type : gpu
Version : OpenCL 3.0 NEO
Name : Intel(R) Data Center GPU Max 1550
Vendor : Intel(R) Corporation
Driver : 25.18.33578
UUID : 8680d50b-2f00-0000-4d00-000000000002
DeviceID : 3029
Num SubDevices : 0
Num SubSubDevices : 0
Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_srgb ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_private_alloca ext_oneapi_atomic16 ext_oneapi_virtual_functions ext_intel_device_info_node_mask
info::device::sub_group_sizes: 16 32
Architecture: intel_gpu_pvc
Device [#2]:
Type : gpu
Version : OpenCL 3.0 NEO
Name : Intel(R) Data Center GPU Max 1550
Vendor : Intel(R) Corporation
Driver : 25.18.33578
UUID : 8680d50b-2f00-0000-9a00-000000000001
DeviceID : 3029
Num SubDevices : 0
Num SubSubDevices : 0
Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_srgb ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_private_alloca ext_oneapi_atomic16 ext_oneapi_virtual_functions ext_intel_device_info_node_mask
info::device::sub_group_sizes: 16 32
Architecture: intel_gpu_pvc
Device [#3]:
Type : gpu
Version : OpenCL 3.0 NEO
Name : Intel(R) Data Center GPU Max 1550
Vendor : Intel(R) Corporation
Driver : 25.18.33578
UUID : 8680d50b-2f00-0000-9a00-000000000002
DeviceID : 3029
Num SubDevices : 0
Num SubSubDevices : 0
Aspects : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_oneapi_srgb ext_intel_gpu_hw_threads_per_eu ext_intel_device_id ext_intel_esimd ext_oneapi_fragment ext_oneapi_chunk ext_oneapi_tangle ext_intel_matrix ext_oneapi_private_alloca ext_oneapi_atomic16 ext_oneapi_virtual_functions ext_intel_device_info_node_mask
info::device::sub_group_sizes: 16 32
Architecture: intel_gpu_pvc
default_selector() : gpu, Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
accelerator_selector() : No device of requested type available.
cpu_selector() : No device of requested type available.
gpu_selector() : gpu, Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
custom_selector(gpu) : gpu, Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Data Center GPU Max 1550 12.60.7 [1.6.33578+42]
custom_selector(cpu) : No device of requested type available.
custom_selector(acc) : No device of requested type available.
[INFO]: unloaded adapter 0x0x608a6aae5210
[INFO]: unloaded adapter 0x0x608a6aae82f0
[INFO]: unloaded adapter 0x0x608a6aaeb6c0
[INFO]: ---> urLoaderTearDown() -> UR_RESULT_SUCCESS
```

### Additional context

For the original reproducer based on pytorch CI test see https://github.com/intel/torch-xpu-ops/issues/4842.

@sarnex, @KornevNikita

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied Python index_add reproduction with dpclang v7.0.0 and compare it with oneAPI 2026.0; the build commands and environment variables are in the issue. Trace the XPU/SYCL compiler path involved in bool index_add, then add or update a regression test for the demonstrated tensor inputs and verify the expected output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.