intel / intel/torch-xpu-ops

[Bug Skip]: UT failures 2026-8-5

Open
#4,825 10 comments 0 reactions 0 assignees View on GitHub
agent:active module: inductor skipped test: ut
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

### 🐛 Describe the bug with skip template

Cases:
~~op_ut,test.dynamo.test_dynamic_shapes.DynamicShapesMiscTests,test_debugmode_respects_custom_op_aliasing_env_override_dynamic_shapes~~
op_ut,test.dynamo.test_dynamic_shapes.DynamicShapesCtxManagerTests,test_autocast_sdpa_dynamic_shapes
op_ut,third_party.torch-xpu-ops.test.xpu.test_linalg_xpu.TestLinalgXPU,test_addmm_out_distinct_c_and_d_is_selected_xpu_bfloat16
op_ut,third_party.torch-xpu-ops.test.xpu.test_linalg_xpu.TestLinalgXPU,test_addmm_out_distinct_c_and_d_is_selected_xpu_float32
~~op_ut,third_party.torch-xpu-ops.test.xpu.test_nn_xpu.TestNNDeviceTypeXPU,test_warp_softmax_64bit_indexing_xpu_float32~~

## ErrorLog
### AssertionError: Scalars are not equal!
```bash
_ DynamicShapesMiscTests.test_debugmode_respects_custom_op_aliasing_env_override_dynamic_shapes _
[gw1] linux -- Python 3.10.20 /tmp/xpu-tool/Python/3.10.20/x64/bin/python
Traceback (most recent call last):
File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/test/dynamo/test_misc.py", line 16799, in test_debugmode_respects_custom_op_aliasing_env_override
self.assertEqual(
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/test_case.py", line 127, in assertEqual
return super().assertEqual(x, y, *args, **kwargs)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4844, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Scalars are not equal!

Expected 0 but got 1.
Absolute difference: 1
Relative difference: inf
stdout:

stderr:
Traceback (most recent call last):
File "", line 15, in
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/__init__.py", line 3278, in compile
backend = _TorchCompileWrapper(backend, mode, options, dynamic)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/__init__.py", line 2985, in __init__
self.compiler_fn = lookup_backend(backend)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/backends/registry.py", line 128, in lookup_backend
_lazy_import()
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/backends/registry.py", line 168, in _lazy_import
import_submodule(backends)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 4568, in import_submodule
importlib.import_module(f"{mod.__name__}.{filename[:-3]}")
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/backends/cudagraphs.py", line 34, in
from torch._dynamo.backends.debugging import boxed_nop
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/backends/debugging.py", line 34, in
from functorch.compile import min_cut_rematerialization_partition
ModuleNotFoundError: No module named 'functorch.compile'
```
### torch._dynamo.exc.InternalTorchDynamoError: NotEqualError: ShapeEnv not equal: field values don't match:
```bash
________ DynamicShapesCtxManagerTests.test_autocast_sdpa_dynamic_shapes ________
[gw3] linux -- Python 3.10.20 /tmp/xpu-tool/Python/3.10.20/x64/bin/python
Traceback (most recent call last):
File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/test/dynamo/test_ctx_manager.py", line 316, in test_autocast_sdpa
compiled = opt_mod(query, key, value)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 563, in __call__
return super().__call__(*args, **kwargs)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1780, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1791, in _call_impl
return forward_call(*args, **kwargs)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1261, in compile_wrapper
result = fn(*args, **kwargs)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1780, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1791, in _call_impl
return forward_call(*args, **kwargs)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 2649, in __call__
result = self._torchdynamo_orig_backend(
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 2333, in __call__
result = self._inner_convert(
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 778, in __call__
result = _compile(
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 2182, in _compile
raise InternalTorchDynamoError(
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 2116, in _compile
guarded_code, tracer_output = compile_inner(code, one_graph, hooks)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_utils_internal.py", line 114, in wrapper_function
return function(*args, **kwargs)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1701, in compile_inner
result = _compile_inner(code, one_graph, hooks)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1917, in _compile_inner
check_fn = dynamo_output.build_guards(
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1013, in build_guards
return CheckFunctionManager(
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 4647, in __init__
builder, guard_manager = self.build_guards(
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 4967, in build_guards
guard.create(builder)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_guards.py", line 373, in create
return self.create_fn(builder, self)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 3460, in SHAPE_ENV
python_code_parts, verbose_code_parts = _get_code_parts(
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 3433, in _get_code_parts
return output_graph.shape_env.produce_guards_verbose(
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 6120, in produce_guards_verbose
self.check_equal(shape_env)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4387, in check_equal
shape_env_check_state_equal(self, other, non_state_variable_names, map_value)
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/fx/experimental/recording.py", line 535, in shape_env_check_state_equal
raise NotEqualError("field values don't match:", errors)
torch._dynamo.exc.InternalTorchDynamoError: NotEqualError: ShapeEnv not equal: field values don't match:

==> guards: values don't match.
> Left: [s83 <= 576, (FloatTrueDiv(1.0, FloatPow(ToFloat(s83), 0.5))) >= 0.0, (FloatTrueDiv(1.0, FloatPow(ToFloat(s83), 0.5))) >= 0.0]
> Right: [s83 <= 576, (FloatTrueDiv(1.0, FloatPow(ToFloat(s83), 0.5))) >= 0.0, (FloatTrueDiv(1.0, FloatPow(ToFloat(s83), 0.5))) >= 0.0, (FloatTrueDiv(1.0, FloatPow(ToFloat(s83), 0.5))) >= 0.0]
```
### AssertionError: Torch not compiled with CUDA enabled
```bash
____ TestLinalgXPU.test_addmm_out_distinct_c_and_d_is_selected_xpu_bfloat16 ____
[gw4] linux -- Python 3.10.20 /tmp/xpu-tool/Python/3.10.20/x64/bin/python
Traceback (most recent call last):
File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_linalg.py", line 6511, in test_addmm_out_distinct_c_and_d_is_selected
self.assertEqual(kernel_count(contiguous), 1)
File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_linalg.py", line 6499, in kernel_count
torch.cuda.synchronize()
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/cuda/__init__.py", line 1275, in synchronize
_lazy_init()
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/cuda/__init__.py", line 580, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

____ TestLinalgXPU.test_addmm_out_distinct_c_and_d_is_selected_xpu_float32 _____
[gw4] linux -- Python 3.10.20 /tmp/xpu-tool/Python/3.10.20/x64/bin/python
Traceback (most recent call last):
File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_linalg.py", line 6511, in test_addmm_out_distinct_c_and_d_is_selected
self.assertEqual(kernel_count(contiguous), 1)
File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/../../../../test/test_linalg.py", line 6499, in kernel_count
torch.cuda.synchronize()
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/cuda/__init__.py", line 1275, in synchronize
_lazy_init()
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/site-packages/torch/cuda/__init__.py", line 580, in _lazy_init
raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
```
### AssertionError: False is not true
```bash
______ TestNNDeviceTypeXPU.test_warp_softmax_64bit_indexing_xpu_float32 _______
[gw7] linux -- Python 3.10.20 /tmp/xpu-tool/Python/3.10.20/x64/bin/python
Traceback (most recent call last):
File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/test_nn_xpu.py", line 14132, in test_warp_softmax_64bit_indexing
run_test(
File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/test_nn_xpu.py", line 14130, in run_test
self.assertTrue(torch.allclose(x.grad.cpu(), xx.grad, rtol=rtol, atol=1e-3))
File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/unittest/case.py", line 687, in assertTrue
raise self.failureException(msg)
AssertionError: False is not true
```
## Pytorch Version
71b66af50dd18422e9ed558d83e3582d14aa1639

### Versions

Detail
Collecting environment information...
PyTorch version: 2.14.0a0+git71b66af
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04.4 LTS (x86_64)
GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version: Could not collect
CMake version: version 3.31.6
Libc version: glibc-2.39

Python version: 3.10.20 (main, Jun 16 2026, 21:01:32) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-6.8.0-136-generic-x86_64-with-glibc2.39
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: True
XPU used to build PyTorch: 20260100
[pip3] tcmlib==1.5.0
[pip3] torch==2.14.0a0+git71b66af
[pip3] torchao==0.19.0.dev20260805+xpu
[pip3] torchaudio==2.11.0a0+4e3e282
[pip3] torchvision==0.29.0a0+10f68db
[pip3] triton-xpu==3.8.0+git1e2d42a0
[pip3] umf==1.1.0
[conda] No relevant packages

Contributor guide

Open the contributing guide

Research direction

Start by running the listed tests in pytorch/test/dynamo/test_misc.py, pytorch/test/dynamo/test_ctx_manager.py, test/test_linalg.py, and test/xpu/test_nn_xpu.py in the reported Python 3.10.20 environment. Compare each failure with its traceback and determine whether the test should be fixed or skipped; done means the remaining cases no longer fail unexpectedly in the XPU test run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.