[PyTorch CI] TestSDPAGpuOnlyXPU.test_scaled_dot_product_attention_fused_kernels_packed_accuracy_type_dense_fused_kernel0_xpu fails: Tensor-likes are not close
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 107
Description
## Description
`test_scaled_dot_product_attention_fused_kernels_packed_accuracy_type_dense_fused_kernel0_xpu` (`TestSDPAGpuOnlyXPU`, `test/test_transformers.py`) fails on XPU with a large accuracy mismatch when running the FLASH_ATTENTION fused kernel (`fused_kernel0` == `SDPBackend.FLASH_ATTENTION`) against the MATH reference backend, for the `dense` (non-nested) input case.
```
AssertionError: Tensor-likes are not close!
Mismatched elements: 32655 / 32768 (99.7%)
Greatest absolute difference: 2.993957042694092 at index (12, 1, 0, 15) (up to 0.007 allowed)
Greatest relative difference: 1.0007356405258179 at index (5, 1, 1, 4) (up to 0.007 allowed)
```
## Root cause / context
This test cell is running on XPU **for the first time** as of pytorch/pytorch#185115 ("[Test] Enable `TestSDPAGpuOnly` on XPU and skip cases unsupported by torch-xpu-ops"), which renamed `TestSDPACudaOnly` -> `TestSDPAGpuOnly` and instantiated it on XPU via `instantiate_device_type_tests`.
That PR already narrowed the `type` parametrize to `["dense"]` only on XPU (skipping `nested`, tracked at #3132), but it did **not** add a skip for the `dense` + `FLASH_ATTENTION` combination, so this cell now runs unguarded and fails with a near-100% element mismatch — i.e. XPU's flash attention fused kernel is producing numerically incorrect output for this dense packed-QKV case, not just failing tolerance by a small margin.
PR #185115's own DrCI AI verdict on the XPU CI failure confirms this analysis:
> "This causes `test_scaled_dot_product_attention_fused_kernels_packed_accuracy` with `type="dense"` and `fused_kernel=FLASH_ATTENTION` to run on XPU for the first time. The test fails because XPU's flash attention kernel produces inaccurate results for this case, which was not skipped by the PR."
No existing tracking issue was found in `intel/torch-xpu-ops` for this specific dense+FLASH_ATTENTION accuracy cell (related but distinct issues: #3132 nested-type skip, #3133 nested-contiguity RuntimeError).
## Failing test
```
Cases:
op_ut,third_party.torch-xpu-ops.test.xpu.test_transformers_xpu.TestSDPAGpuOnlyXPU,test_scaled_dot_product_attention_fused_kernels_packed_accuracy_type_dense_fused_kernel0_xpu
```
## Reproduction
```
python test/test_transformers.py TestSDPAGpuOnlyXPU.test_scaled_dot_product_attention_fused_kernels_packed_accuracy_type_dense_fused_kernel0_xpu
```
## CI reference
- PR: https://github.com/pytorch/pytorch/pull/185115
- Failing job: https://github.com/pytorch/pytorch/actions/runs/29566643723/job/88044561469
- Related: #3132, #3133
## Action items
- [ ] Confirm root cause (XPU flash attention kernel numerical bug vs. missing skip)
- [ ] Add `@skipIfXpu` / in-body skip for `type="dense"` + `fused_kernel=FLASH_ATTENTION` in `test_scaled_dot_product_attention_fused_kernels_packed_accuracy` in `pytorch/pytorch` `test/test_transformers.py`, referencing this issue
- [ ] Mirror the skip in `third_party/torch-xpu-ops/test/xpu/skip_list_common.py`
- [ ] Investigate/fix the underlying XPU flash attention kernel accuracy issue for dense packed QKV inputs
Contributor guide
Research direction
Start with test/test_transformers.py and reproduce the named TestSDPAGpuOnlyXPU case using the command in the issue. Compare the dense FLASH_ATTENTION result with the MATH reference, then inspect third_party/torch-xpu-ops/test/xpu/skip_list_common.py and the related issues. Done means the failure is explained and the appropriate upstream and mirrored skip, or the underlying kernel fix, is covered by the relevant test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- backend, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100