[BMG] TestLearnableBiasesXPU: Compiled FlexAttention is less accurate than eager in fp32 (45 test cases, device-agnostic)
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 13h
- Merged PRs (30d)
- 107
Description
## Description
45 parametrized cases of `TestLearnableBiasesXPU` (`test/inductor/test_flex_attention.py`, various bias-type/batch/head/seq_len/headdim variants) are pre-emptively SKIPPED on XPU with the identical message:
```
Compiled FlexAttention is less accurate than eager in fp32
```
## Root cause investigation
- The skip fires from a runtime accuracy check in `_gold_check` (`test_flex_attention.py:8176-8181`): when the compiled-vs-eager error ratio exceeds tolerance in fp32, the test calls `self.skipTest(...)` rather than asserting/failing.
- **Confirmed device-agnostic**: the skip condition is gated on `dtype == torch.float32`, not on `self.device == "xpu"` or any device-specific check. This means the same skip would fire identically on CUDA/CPU if the same fp32 accuracy gap is hit there — this is a general Compiled-FlexAttention fp32 precision limitation, not an XPU-specific enablement gap.
- Base class `TestLearnableBiases` confirmed present (`test_flex_attention.py:8141`), instantiated with `allow_xpu=True` (`test_flex_attention.py:9288`) — not a community change.
- No tracking issue found in `pytorch/pytorch` or `intel/torch-xpu-ops` for this specific skip message.
## Why file this (despite being device-agnostic)
No existing issue tracks this cross-backend fp32 precision gap in Compiled FlexAttention. Filing so the limitation is documented and the 45 currently-skipped XPU test cases have a reference for future re-evaluation.
## Reproduction
```
python -m pytest "test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_absolute_2d_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_cuda" -v
```
## Environment
- torch: 2.13.0+xpu
- Sheet: `XPU skipped only Inductor`, 45 rows in `release-2.13-bmg-2.xlsx`
- Cross-reference: may share a root cause with the FlexAttention/PagedAttention accuracy issue filed separately (same file, same general "compiled FlexAttention accuracy" theme, though this skip is confirmed device-agnostic while the other issue's failures are XPU test-suite observations without a confirmed CUDA comparison).
Contributor guide
Research direction
Start in test/inductor/test_flex_attention.py, especially _gold_check at lines 8176-8181 and the TestLearnableBiasesXPU cases around line 9288. Run the supplied pytest reproduction to confirm the fp32 accuracy skip. Done means the device-agnostic compiled-versus-eager accuracy limitation has a tracked resolution or a basis for re-evaluating the 45 skipped cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100