intel / intel/torch-xpu-ops

New failures in test_scaled_matmul_cuda_xpu.py

Open
#5,339 1 comment 0 reactions 0 assignees View on GitHub
skipped_bmg
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

### 🐛 Describe the bug

Cases:
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_contraction_dim_fake_False_inplace_False_contraction_dim0_supported_True_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_contraction_dim_fake_False_inplace_False_contraction_dim1_supported_True_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_contraction_dim_fake_False_inplace_True_contraction_dim0_supported_True_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_contraction_dim_fake_False_inplace_True_contraction_dim1_supported_True_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_cudagraph_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_empty_inplace_False_m_0_n_32_k_16_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_empty_inplace_False_m_32_n_0_k_16_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_empty_inplace_False_m_32_n_32_k_0_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_empty_inplace_True_m_0_n_32_k_16_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_empty_inplace_True_m_32_n_0_k_16_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_empty_inplace_True_m_32_n_32_k_0_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_fullgraph_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_mxfp8_bfloat16_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_mxfp8_float16_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_mxfp8_float32_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_nvfp4_two_level_False_bfloat16_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_nvfp4_two_level_False_float16_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_nvfp4_two_level_False_float32_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_nvfp4_two_level_True_bfloat16_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_nvfp4_two_level_True_float16_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_nvfp4_two_level_True_float32_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_scalar_semantics_inplace_False_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_scalar_semantics_inplace_True_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_tensorwise_bfloat16_m_1_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_tensorwise_bfloat16_m_64_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_tensorwise_float16_m_1_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_tensorwise_float16_m_64_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_tensorwise_float32_m_1_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_tensorwise_float32_m_64_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_validation_xpu
op_ut,third_party.torch-xpu-ops.test.xpu.test_scaled_matmul_cuda_xpu.TestFP8MatmulXPU,test_scaled_addmm_wgrad_accumulation_xpu

## ErrorLog
### NotImplementedError: The operator 'aten::_scaled_addmm.out' is not currently implemented for the XPU device.
```bash
_______________ TestFP8MatmulXPU.test_scaled_addmm_fullgraph_xpu _______________
[gw6] linux -- Python 3.10.21 /__w/torch-xpu-ops/torch-xpu-ops/.venv/bin/python
Traceback (most recent call last):
File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scaled_matmul_cuda.py", line 1586, in test_scaled_addmm_fullgraph
expected = fn(input, *operands)
File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scaled_matmul_cuda.py", line 1581, in fn
return scaled_addmm(
File "/__w/torch-xpu-ops/torch-xpu-ops/.venv/lib/python3.10/site-packages/torch/nn/functional.py", line 7325, in scaled_addmm
return torch._scaled_addmm(
NotImplementedError: The operator 'aten::_scaled_addmm.out' is not currently implemented for the XPU device. Please open a feature on https://github.com/intel/torch-xpu-ops/issues. You can set the environment variable `PYTORCH_ENABLE_XPU_FALLBACK=1` to use the CPU implementation as a fallback for XPU unimplemented operators. WARNING: this will bring unexpected performance compared with running natively on XPU.
```
### ValueError: For XPU MX/NVFP4 gemm, scale_a swizzle entries must all be NO_SWIZZLE
```bash
____________ TestFP8MatmulXPU.test_scaled_addmm_mxfp8_bfloat16_xpu _____________
[gw6] linux -- Python 3.10.21 /__w/torch-xpu-ops/torch-xpu-ops/.venv/bin/python
Traceback (most recent call last):
File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_scaled_matmul_cuda.py", line 1480, in test_scaled_addmm_mxfp8
actual = scaled_addmm(input, *args, **kwargs)
File "/__w/torch-xpu-ops/torch-xpu-ops/.venv/lib/python3.10/site-packages/torch/nn/functional.py", line 7325, in scaled_addmm
return torch._scaled_addmm(
ValueError: For XPU MX/NVFP4 gemm, scale_a swizzle entries must all be NO_SWIZZLE
```
## Pytorch Version
[8374e0f](https://github.com/pytorch/pytorch/commit/8374e0fbb472a88cb5be7fcf0f14d5b3f8ec08ac)
## Torch-xpu-ops
c513b6ad85ffe36858bfbf39fb1103d19cecd8e2

### Versions

Detail
Collecting environment information...
PyTorch version: 2.15.0a0+git8374e0f
Is debug build: False
CUDA used to build PyTorch: None
ROCm SDK used to build PyTorch: N/A
HIP used to build PyTorch: N/A

OS: Ubuntu 26.04 LTS (x86_64)
GCC version: (Ubuntu 13.4.0-10ubuntu1) 13.4.0
Clang version: Could not collect
CMake version: version 3.31.6
Libc version: glibc-2.43

Python version: 3.10.21 (main, Sep 1 2026, 14:16:49) [Clang 22.1.3 ] (64-bit runtime)
Python platform: Linux-7.0.0-14-generic-x86_64-with-glibc2.43
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: True
XPU used to build PyTorch: 20260100
Intel GPU driver version:
* intel-opencl-icd: 26.18.38308.1-1~26.04~ppa1
* libze1: 1.28.2-2
Intel GPU models onboard:
* Intel(R) Arc(TM) Pro B60 Graphics
* Intel(R) Arc(TM) Pro B60 Graphics
* Intel(R) Arc(TM) Pro B60 Graphics
* Intel(R) Arc(TM) Pro B60 Graphics
* Intel(R) Arc(TM) Pro B60 Graphics
* Intel(R) Arc(TM) Pro B60 Graphics
* Intel(R) Arc(TM) Pro B60 Graphics
* Intel(R) Arc(TM) Pro B60 Graphics
Intel GPU models detected:
* [0] _XpuDeviceProperties(name='Intel(R) Arc(TM) Pro B60 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE211, uuid=868011e2-0000-0000-1700-000000000000, driver_version='1.15.38308+1', total_memory=24480MB, local_mem_size=128KB, last_level_cache_size=18432KB, max_compute_units=160, memory_clock_rate=2400MHz, memory_bus_width=64-bit, gpu_eu_count=160, gpu_subslice_count=20, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
* [1] _XpuDeviceProperties(name='Intel(R) Arc(TM) Pro B60 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE211, uuid=868011e2-0000-0000-2c00-000000000000, driver_version='1.15.38308+1', total_memory=24480MB, local_mem_size=128KB, last_level_cache_size=18432KB, max_compute_units=160, memory_clock_rate=2400MHz, memory_bus_width=64-bit, gpu_eu_count=160, gpu_subslice_count=20, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
* [2] _XpuDeviceProperties(name='Intel(R) Arc(TM) Pro B60 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE211, uuid=868011e2-0000-0000-3d00-000000000000, driver_version='1.15.38308+1', total_memory=24480MB, local_mem_size=128KB, last_level_cache_size=18432KB, max_compute_units=160, memory_clock_rate=2400MHz, memory_bus_width=64-bit, gpu_eu_count=160, gpu_subslice_count=20, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
* [3] _XpuDeviceProperties(name='Intel(R) Arc(TM) Pro B60 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE211, uuid=868011e2-0000-0000-4e00-000000000000, driver_version='1.15.38308+1', total_memory=24480MB, local_mem_size=128KB, last_level_cache_size=18432KB, max_compute_units=160, memory_clock_rate=2400MHz, memory_bus_width=64-bit, gpu_eu_count=160, gpu_subslice_count=20, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
* [4] _XpuDeviceProperties(name='Intel(R) Arc(TM) Pro B60 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE211, uuid=868011e2-0000-0000-9700-000000000000, driver_version='1.15.38308+1', total_memory=24480MB, local_mem_size=128KB, last_level_cache_size=18432KB, max_compute_units=160, memory_clock_rate=2400MHz, memory_bus_width=64-bit, gpu_eu_count=160, gpu_subslice_count=20, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
* [5] _XpuDeviceProperties(name='Intel(R) Arc(TM) Pro B60 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE211, uuid=868011e2-0000-0000-a900-000000000000, driver_version='1.15.38308+1', total_memory=24480MB, local_mem_size=128KB, last_level_cache_size=18432KB, max_compute_units=160, memory_clock_rate=2400MHz, memory_bus_width=64-bit, gpu_eu_count=160, gpu_subslice_count=20, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
* [6] _XpuDeviceProperties(name='Intel(R) Arc(TM) Pro B60 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE211, uuid=868011e2-0000-0000-ba00-000000000000, driver_version='1.15.38308+1', total_memory=24480MB, local_mem_size=128KB, last_level_cache_size=18432KB, max_compute_units=160, memory_clock_rate=2400MHz, memory_bus_width=64-bit, gpu_eu_count=160, gpu_subslice_count=20, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
* [7] _XpuDeviceProperties(name='Intel(R) Arc(TM) Pro B60 Graphics', platform_name='Intel(R) oneAPI Unified Runtime over Level-Zero V2', type='gpu', device_id=0xE211, uuid=868011e2-0000-0000-cb00-000000000000, driver_version='1.15.38308+1', total_memory=24480MB, local_mem_size=128KB, last_level_cache_size=18432KB, max_compute_units=160, memory_clock_rate=2400MHz, memory_bus_width=64-bit, gpu_eu_count=160, gpu_subslice_count=20, max_work_group_size=1024, max_num_sub_groups=64, sub_group_sizes=[16 32], has_fp16=1, has_fp64=1, has_atomic64=1, is_integrated_gpu=0)
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False
Caching allocator config: N/A

Vulnerability Vmscape: Mitigation; IBPB before exit to userspace

Versions of relevant libraries:
[pip3] dpcpp-cpp-rt==2026.1.0
[pip3] impi-rt==2021.18.1
[pip3] intel-cmplr-lib-rt==2026.1.0
[pip3] intel-cmplr-lib-ur==2026.1.0
[pip3] intel-cmplr-lic-rt==2026.1.0
[pip3] intel-opencl-rt==2026.1.0
[pip3] intel-openmp==2026.1.0
[pip3] intel-pti==1.0.1
[pip3] intel-sycl-rt==2026.1.0
[pip3] mkl==2026.1.0
[pip3] mypy==1.16.0
[pip3] mypy_extensions==1.1.0
[pip3] numpy==1.24.4
[pip3] nvidia-cuda-cupti==13.4.58
[pip3] oneccl==2022.1.1
[pip3] oneccl-devel==2022.1.1
[pip3] onemkl-license==2026.1.0
[pip3] onemkl-sycl-blas==2026.1.0
[pip3] onemkl-sycl-dft==2026.1.0
[pip3] onemkl-sycl-lapack==2026.1.0
[pip3] onemkl-sycl-rng==2026.1.0
[pip3] onemkl-sycl-sparse==2026.1.0
[pip3] onnx==1.21.0
[pip3] onnx-ir==0.1.16
[pip3] onnxscript==0.6.2
[pip3] optree==0.13.0
[pip3] tbb==2023.1.0
[pip3] tcmlib==1.5.0
[pip3] torch==2.15.0a0+git8374e0f
[pip3] torchao==0.19.0.dev20260911+xpu
[pip3] torchaudio==2.11.0a0+b85c99c
[pip3] torchvision==0.30.0a0+ac8d215
[pip3] triton-xpu==3.8.0+git1e2d42a0
[pip3] umf==1.1.0
[conda] No relevant packages

Contributor guide

Open the contributing guide

Research direction

Start by running the listed TestFP8MatmulXPU failures and read test_scaled_matmul_cuda.py around lines 1480 and 1581-1586. Trace the XPU handling for aten::_scaled_addmm.out and the MX/NVFP4 scale validation. Done means the listed tests pass natively on XPU without enabling CPU fallback.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.