intel / intel/torch-xpu-ops

[PyTorch CI] TestSDPAGpuOnlyXPU.test_scaled_dot_product_attention_fused_kernels_packed_accuracy_type_dense_fused_kernel0_xpu fails: Tensor-likes are not close

Open
#4,401 4 comments 0 reactions 0 assignees View on GitHub
agent:needs-human bug pytorch-ci-failure test: ut
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

## Description

`test_scaled_dot_product_attention_fused_kernels_packed_accuracy_type_dense_fused_kernel0_xpu` (`TestSDPAGpuOnlyXPU`, `test/test_transformers.py`) fails on XPU with a large accuracy mismatch when running the FLASH_ATTENTION fused kernel (`fused_kernel0` == `SDPBackend.FLASH_ATTENTION`) against the MATH reference backend, for the `dense` (non-nested) input case.

```
AssertionError: Tensor-likes are not close!

Mismatched elements: 32655 / 32768 (99.7%)
Greatest absolute difference: 2.993957042694092 at index (12, 1, 0, 15) (up to 0.007 allowed)
Greatest relative difference: 1.0007356405258179 at index (5, 1, 1, 4) (up to 0.007 allowed)
```

## Root cause / context

This test cell is running on XPU **for the first time** as of pytorch/pytorch#185115 ("[Test] Enable `TestSDPAGpuOnly` on XPU and skip cases unsupported by torch-xpu-ops"), which renamed `TestSDPACudaOnly` -> `TestSDPAGpuOnly` and instantiated it on XPU via `instantiate_device_type_tests`.

That PR already narrowed the `type` parametrize to `["dense"]` only on XPU (skipping `nested`, tracked at #3132), but it did **not** add a skip for the `dense` + `FLASH_ATTENTION` combination, so this cell now runs unguarded and fails with a near-100% element mismatch — i.e. XPU's flash attention fused kernel is producing numerically incorrect output for this dense packed-QKV case, not just failing tolerance by a small margin.

PR #185115's own DrCI AI verdict on the XPU CI failure confirms this analysis:

> "This causes `test_scaled_dot_product_attention_fused_kernels_packed_accuracy` with `type="dense"` and `fused_kernel=FLASH_ATTENTION` to run on XPU for the first time. The test fails because XPU's flash attention kernel produces inaccurate results for this case, which was not skipped by the PR."

No existing tracking issue was found in `intel/torch-xpu-ops` for this specific dense+FLASH_ATTENTION accuracy cell (related but distinct issues: #3132 nested-type skip, #3133 nested-contiguity RuntimeError).

## Failing test

```
Cases:
op_ut,third_party.torch-xpu-ops.test.xpu.test_transformers_xpu.TestSDPAGpuOnlyXPU,test_scaled_dot_product_attention_fused_kernels_packed_accuracy_type_dense_fused_kernel0_xpu
```

## Reproduction

```
python test/test_transformers.py TestSDPAGpuOnlyXPU.test_scaled_dot_product_attention_fused_kernels_packed_accuracy_type_dense_fused_kernel0_xpu
```

## CI reference

- PR: https://github.com/pytorch/pytorch/pull/185115
- Failing job: https://github.com/pytorch/pytorch/actions/runs/29566643723/job/88044561469
- Related: #3132, #3133

## Action items

- [ ] Confirm root cause (XPU flash attention kernel numerical bug vs. missing skip)
- [ ] Add `@skipIfXpu` / in-body skip for `type="dense"` + `fused_kernel=FLASH_ATTENTION` in `test_scaled_dot_product_attention_fused_kernels_packed_accuracy` in `pytorch/pytorch` `test/test_transformers.py`, referencing this issue
- [ ] Mirror the skip in `third_party/torch-xpu-ops/test/xpu/skip_list_common.py`
- [ ] Investigate/fix the underlying XPU flash attention kernel accuracy issue for dense packed QKV inputs

Contributor guide

Open the contributing guide

Research direction

Start with test/test_transformers.py and reproduce the named TestSDPAGpuOnlyXPU case using the command in the issue. Compare the dense FLASH_ATTENTION result with the MATH reference, then inspect third_party/torch-xpu-ops/test/xpu/skip_list_common.py and the related issues. Done means the failure is explained and the appropriate upstream and mirrored skip, or the underlying kernel fix, is covered by the relevant test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
backend, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.