intel / intel/torch-xpu-ops

[BMG] TestFlexAttentionXPU/TestPagedAttentionXPU: compiled-vs-eager accuracy tolerance exceeded across many parametrized variants (71 test cases)

Open
#4,235 0 comments 0 reactions 0 assignees View on GitHub
bug module: inductor test: ut
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

## Description

63 parametrized cases of `TestFlexAttentionXPU` (`test/inductor/test_flex_attention.py`, `test_builtin_score_mods_*` variants across `score_mod`/`BLOCK_SIZE`/`head_dims`) and 8 parametrized cases of `TestPagedAttentionXPU` (`test_paged_builtin_score_mods_*`) fail on XPU with the same accuracy-tolerance error template:

```
AssertionError: [False is not true : ][Out ]Compiled error X is greater than ref error Y by more than Z.ZX.
```

i.e. `torch.compile`'d FlexAttention output diverges from the eager reference by more than the allowed tolerance multiplier, across many parametrized `score_mod`, `BLOCK_SIZE`, and `head_dims` variants in both the standard and paged-attention test classes.

## Root cause investigation

- Both classes' base methods are confirmed present and unmodified upstream (not a community change):
- `TestFlexAttentionXPU`: base method `test_builtin_score_mods_automatic_dynamic` at `test_flex_attention.py:1539` (commit `dfab69fdf122`, ancestor of HEAD); `test_score_mods` array confirmed with 8 elements (commit `762ce6f062f3`, ancestor of HEAD).
- `TestPagedAttentionXPU`: base method `test_paged_builtin_score_mods` at `test_flex_attention.py:7996`, instantiated with `allow_xpu=True` + `@dtypesIfXPU`.
- No open tracking issue found. Two closed candidates were checked and ruled out:
- `intel/torch-xpu-ops#1444` was withdrawn by its own author ("sorry for creating the incorrect issue").
- `intel/torch-xpu-ops#1521` tracked a different, already-resolved bug (a basic device-detection failure, `Torch not compiled with CUDA enabled`), closed when FlexAttention-for-XPU support landed upstream. Not the same accuracy issue.
- The two classes are grouped into one issue because they share the identical error template and file, strongly suggesting the same underlying numerics gap in the compiled FlexAttention/paged-attention kernel on XPU.

## Sample failures

```
test_builtin_score_mods_score_mod0_cuda_float32:
Out Compiled error 3.234612694903627e-05 is greater than ref error 2.983891779817198e-08 by more than 10.0X.

test_paged_builtin_score_mods_score_mod0_cuda_float32:
Out Compiled error 4.1584374462482134e-05 is greater than ref error 2.5364025013666975e-08 by more than 10.0X.
```

## Environment

- torch: 2.13.0+xpu
- Sheet: `XPU skipped only Inductor`, 71 rows total (63 + 8) in `release-2.13-bmg-2.xlsx`
- Note: `TestLearnableBiasesXPU` (separate issue, see cross-reference) shows a related but structurally distinct symptom (pre-emptive skip on a device-agnostic fp32 accuracy check) in the same file — may share a root cause with this issue; worth cross-checking during investigation.

Contributor guide

Open the contributing guide

Research direction

Start by running the parametrized TestFlexAttentionXPU and TestPagedAttentionXPU cases in test/inductor/test_flex_attention.py, including the score_mod, BLOCK_SIZE, and head_dims variants. Read the base methods at lines 1539 and 7996 and compare compiled outputs with eager references on XPU. Done means the 71 cases no longer exceed their accuracy tolerances, with any remaining failures explained and documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.