[BMG] TestFlexAttentionXPU/TestPagedAttentionXPU: compiled-vs-eager accuracy tolerance exceeded across many parametrized variants (71 test cases)
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
## Description
63 parametrized cases of `TestFlexAttentionXPU` (`test/inductor/test_flex_attention.py`, `test_builtin_score_mods_*` variants across `score_mod`/`BLOCK_SIZE`/`head_dims`) and 8 parametrized cases of `TestPagedAttentionXPU` (`test_paged_builtin_score_mods_*`) fail on XPU with the same accuracy-tolerance error template:
```
AssertionError: [False is not true : ][Out ]Compiled error X is greater than ref error Y by more than Z.ZX.
```
i.e. `torch.compile`'d FlexAttention output diverges from the eager reference by more than the allowed tolerance multiplier, across many parametrized `score_mod`, `BLOCK_SIZE`, and `head_dims` variants in both the standard and paged-attention test classes.
## Root cause investigation
- Both classes' base methods are confirmed present and unmodified upstream (not a community change):
- `TestFlexAttentionXPU`: base method `test_builtin_score_mods_automatic_dynamic` at `test_flex_attention.py:1539` (commit `dfab69fdf122`, ancestor of HEAD); `test_score_mods` array confirmed with 8 elements (commit `762ce6f062f3`, ancestor of HEAD).
- `TestPagedAttentionXPU`: base method `test_paged_builtin_score_mods` at `test_flex_attention.py:7996`, instantiated with `allow_xpu=True` + `@dtypesIfXPU`.
- No open tracking issue found. Two closed candidates were checked and ruled out:
- `intel/torch-xpu-ops#1444` was withdrawn by its own author ("sorry for creating the incorrect issue").
- `intel/torch-xpu-ops#1521` tracked a different, already-resolved bug (a basic device-detection failure, `Torch not compiled with CUDA enabled`), closed when FlexAttention-for-XPU support landed upstream. Not the same accuracy issue.
- The two classes are grouped into one issue because they share the identical error template and file, strongly suggesting the same underlying numerics gap in the compiled FlexAttention/paged-attention kernel on XPU.
## Sample failures
```
test_builtin_score_mods_score_mod0_cuda_float32:
Out Compiled error 3.234612694903627e-05 is greater than ref error 2.983891779817198e-08 by more than 10.0X.
test_paged_builtin_score_mods_score_mod0_cuda_float32:
Out Compiled error 4.1584374462482134e-05 is greater than ref error 2.5364025013666975e-08 by more than 10.0X.
```
## Environment
- torch: 2.13.0+xpu
- Sheet: `XPU skipped only Inductor`, 71 rows total (63 + 8) in `release-2.13-bmg-2.xlsx`
- Note: `TestLearnableBiasesXPU` (separate issue, see cross-reference) shows a related but structurally distinct symptom (pre-emptive skip on a device-agnostic fp32 accuracy check) in the same file — may share a root cause with this issue; worth cross-checking during investigation.
Contributor guide
Research direction
Start by running the parametrized TestFlexAttentionXPU and TestPagedAttentionXPU cases in test/inductor/test_flex_attention.py, including the score_mod, BLOCK_SIZE, and head_dims variants. Read the base methods at lines 1539 and 7996 and compare compiled outputs with eager references on XPU. Done means the 71 cases no longer exceed their accuracy tolerances, with any remaining failures explained and documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100