intel / intel/torch-xpu-ops

[BMG] TestLearnableBiasesXPU: Compiled FlexAttention is less accurate than eager in fp32 (45 test cases, device-agnostic)

Open
#4,236 0 comments 0 reactions 0 assignees View on GitHub
module: inductor test: ut
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

## Description

45 parametrized cases of `TestLearnableBiasesXPU` (`test/inductor/test_flex_attention.py`, various bias-type/batch/head/seq_len/headdim variants) are pre-emptively SKIPPED on XPU with the identical message:

```
Compiled FlexAttention is less accurate than eager in fp32
```

## Root cause investigation

- The skip fires from a runtime accuracy check in `_gold_check` (`test_flex_attention.py:8176-8181`): when the compiled-vs-eager error ratio exceeds tolerance in fp32, the test calls `self.skipTest(...)` rather than asserting/failing.
- **Confirmed device-agnostic**: the skip condition is gated on `dtype == torch.float32`, not on `self.device == "xpu"` or any device-specific check. This means the same skip would fire identically on CUDA/CPU if the same fp32 accuracy gap is hit there — this is a general Compiled-FlexAttention fp32 precision limitation, not an XPU-specific enablement gap.
- Base class `TestLearnableBiases` confirmed present (`test_flex_attention.py:8141`), instantiated with `allow_xpu=True` (`test_flex_attention.py:9288`) — not a community change.
- No tracking issue found in `pytorch/pytorch` or `intel/torch-xpu-ops` for this specific skip message.

## Why file this (despite being device-agnostic)

No existing issue tracks this cross-backend fp32 precision gap in Compiled FlexAttention. Filing so the limitation is documented and the 45 currently-skipped XPU test cases have a reference for future re-evaluation.

## Reproduction

```
python -m pytest "test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_absolute_2d_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_cuda" -v
```

## Environment

- torch: 2.13.0+xpu
- Sheet: `XPU skipped only Inductor`, 45 rows in `release-2.13-bmg-2.xlsx`
- Cross-reference: may share a root cause with the FlexAttention/PagedAttention accuracy issue filed separately (same file, same general "compiled FlexAttention accuracy" theme, though this skip is confirmed device-agnostic while the other issue's failures are XPU test-suite observations without a confirmed CUDA comparison).

Contributor guide

Open the contributing guide

Research direction

Start in test/inductor/test_flex_attention.py, especially _gold_check at lines 8176-8181 and the TestLearnableBiasesXPU cases around line 9288. Run the supplied pytest reproduction to confirm the fp32 accuracy skip. Done means the device-agnostic compiled-versus-eager accuracy limitation has a tracked resolution or a basis for re-evaluating the 45 skipped cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.