intel / intel/torch-xpu-ops

[B60][stock PyTorch][XPU] FlexAttention/PagedAttention accuracy regressions vs PVC

Open
#4,792 4 comments 0 reactions 1 assignee Claimed by @hoshibara View on GitHub
hw: BMG test: ut
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 13h
Merged PRs (30d)
107

Description

## Summary

B60 shows a broad XPU FlexAttention/PagedAttention accuracy regression in stock PyTorch tests. The same tests passed on PVC.

## Failure Category

Numerical accuracy mismatch between compiled output and eager/reference output.

Representative messages:

```text
Compiled FlexAttention is less accurate than eager in fp32
AssertionError: Out Compiled error ... is greater than ref error ... by more than 10.0X
AssertionError: Grad_Query Compiled error ... is greater than ref error ... by more than 10.0X
```

## Scope

- Test suite: stock PyTorch
- Platform comparison: B60 failed or skipped; PVC passed
- Affected rows observed: 117
- Main file: `test/inductor/test_flex_attention.py`

## Representative Failures

```text
test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_automatic_dynamic_score_mod7_xpu_float16
AssertionError: Compiled error 0.00022216421449210188 is greater than ref error 8.428077824380299e-05 by more than 1.1X

test/inductor/test_flex_attention.py::TestPagedAttentionXPU::test_paged_builtin_score_mods_score_mod1_xpu_float32
AssertionError: False is not true : Out Compiled error 0.00012370378724345263 is greater than ref error 7.14636960574454e-08 by more than 10.0X

test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_direct_backward_preserves_explicit_buffers_xpu
AssertionError: Grad_Query Compiled error 6.649729377544006e-05 is greater than ref error 3.495429805113324e-08 by more than 10.0X

test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_absolute_2d_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_xpu
Skipped: Compiled FlexAttention is less accurate than eager in fp32
```

## Expected Behavior

Compiled XPU FlexAttention/PagedAttention should stay within the expected tolerance and match the PVC result.

## Notes

The failures cover many score modifiers, block sizes, head dimensions, and PagedAttention variants, so this looks like a shared FlexAttention numerical accuracy issue rather than isolated individual tests.

## Full Case List

Total cases: 117

Show all 117 cases

1. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_automatic_dynamic_score_mod7_xpu_float16` | `failed`
- `AssertionError: Compiled error 0.00022216421449210188 is greater than ref error 8.428077824380299e-05 by more than 1.1X.`
2. `stock pytorch` | `test/inductor/test_flex_attention.py::TestPagedAttentionXPU::test_paged_builtin_score_mods_score_mod1_xpu_float32` | `failed`
- `AssertionError: False is not true : Out Compiled error 0.00012370378724345263 is greater than ref error 7.14636960574454e-08 by more than 10.0X.`
3. `stock pytorch` | `test/inductor/test_flex_attention.py::TestPagedAttentionXPU::test_paged_builtin_score_mods_score_mod5_xpu_float32` | `failed`
- `AssertionError: False is not true : Out Compiled error 0.00013912540333810858 is greater than ref error 1.4032223513966372e-06 by more than 10.0X.`
4. `stock pytorch` | `test/inductor/test_flex_attention.py::TestPagedAttentionXPU::test_paged_builtin_score_mods_score_mod6_xpu_float32` | `failed`
- `AssertionError: False is not true : Out Compiled error 0.00013912540333810858 is greater than ref error 1.4032223513966372e-06 by more than 10.0X.`
5. `stock pytorch` | `test/inductor/test_flex_attention.py::TestPagedAttentionXPU::test_paged_builtin_score_mods_score_mod4_xpu_float32` | `failed`
- `AssertionError: False is not true : Out Compiled error 0.00014006391064692262 is greater than ref error 0.0 by more than 10.0X.`
6. `stock pytorch` | `test/inductor/test_flex_attention.py::TestPagedAttentionXPU::test_paged_builtin_score_mods_score_mod2_xpu_float32` | `failed`
- `AssertionError: False is not true : Out Compiled error 0.0003554533227057903 is greater than ref error 2.0544184492983257e-07 by more than 10.0X.`
7. `stock pytorch` | `test/inductor/test_flex_attention.py::TestPagedAttentionXPU::test_paged_builtin_score_mods_score_mod0_xpu_float32` | `failed`
- `AssertionError: False is not true : Out Compiled error 4.1584374462482134e-05 is greater than ref error 2.5364025013666975e-08 by more than 10.0X.`
8. `stock pytorch` | `test/inductor/test_flex_attention.py::TestPagedAttentionXPU::test_paged_builtin_score_mods_score_mod3_xpu_float32` | `failed`
- `AssertionError: False is not true : Out Compiled error 4.1584374462482134e-05 is greater than ref error 2.5364025013666975e-08 by more than 10.0X.`
9. `stock pytorch` | `test/inductor/test_flex_attention.py::TestPagedAttentionXPU::test_paged_builtin_score_mods_score_mod7_xpu_float32` | `failed`
- `AssertionError: False is not true : Out Compiled error 9.878438259098405e-05 is greater than ref error 3.202989102634225e-08 by more than 10.0X.`
10. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_direct_backward_preserves_explicit_buffers_xpu` | `failed`
- `AssertionError: Grad_Query Compiled error 6.649729377544006e-05 is greater than ref error 3.495429805113324e-08 by more than 10.0X.`
11. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod1_BLOCK_SIZE2_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00010885315801661213 is greater than ref error 1.1360814057759137e-07 by more than 10.0X.`
12. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod1_BLOCK_SIZE3_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00010885315801661213 is greater than ref error 1.1360814057759137e-07 by more than 10.0X.`
13. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod1_BLOCK_SIZE_128_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00010885315801661213 is greater than ref error 1.1360814057759137e-07 by more than 10.0X.`
14. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod1_BLOCK_SIZE_256_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00010885315801661213 is greater than ref error 1.1360814057759137e-07 by more than 10.0X.`
15. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_score_mod1_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00010885315801661213 is greater than ref error 1.1360814057759137e-07 by more than 10.0X.`
16. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod1_head_dims0_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00010931279921425544 is greater than ref error 1.0742772399304324e-07 by more than 10.0X.`
17. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod1_head_dims1_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0001095148928733713 is greater than ref error 9.017244821750285e-08 by more than 10.0X.`
18. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_function_composition_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00013352268567677777 is greater than ref error 6.806583943896025e-08 by more than 10.0X.`
19. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod5_head_dims1_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00013726300831214952 is greater than ref error 7.093169943542733e-07 by more than 10.0X.`
20. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod6_head_dims1_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0001373512983196341 is greater than ref error 7.083683366229169e-07 by more than 10.0X.`
21. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod5_BLOCK_SIZE2_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00013851325804270016 is greater than ref error 7.145564232611141e-07 by more than 10.0X.`
22. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod5_BLOCK_SIZE3_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00013851325804270016 is greater than ref error 7.145564232611141e-07 by more than 10.0X.`
23. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod5_BLOCK_SIZE_128_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00013851325804270016 is greater than ref error 7.145564232611141e-07 by more than 10.0X.`
24. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod5_BLOCK_SIZE_256_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00013851325804270016 is greater than ref error 7.145564232611141e-07 by more than 10.0X.`
25. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_score_mod5_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00013851325804270016 is greater than ref error 7.145564232611141e-07 by more than 10.0X.`
26. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod6_BLOCK_SIZE2_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0001386839097766206 is greater than ref error 7.135756912538625e-07 by more than 10.0X.`
27. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod6_BLOCK_SIZE3_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0001386839097766206 is greater than ref error 7.135756912538625e-07 by more than 10.0X.`
28. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod6_BLOCK_SIZE_128_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0001386839097766206 is greater than ref error 7.135756912538625e-07 by more than 10.0X.`
29. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod6_BLOCK_SIZE_256_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0001386839097766206 is greater than ref error 7.135756912538625e-07 by more than 10.0X.`
30. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_score_mod6_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0001386839097766206 is greater than ref error 7.135756912538625e-07 by more than 10.0X.`
31. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod5_head_dims0_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00014261869621620067 is greater than ref error 7.462165866999293e-07 by more than 10.0X.`
32. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod6_head_dims0_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.00014279713187634815 is greater than ref error 7.451884506752084e-07 by more than 10.0X.`
33. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod2_head_dims1_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0003365337939377367 is greater than ref error 2.913263697519757e-07 by more than 10.0X.`
34. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod2_BLOCK_SIZE2_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0003401536486123356 is greater than ref error 3.9100063715395213e-07 by more than 10.0X.`
35. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod2_BLOCK_SIZE3_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0003401536486123356 is greater than ref error 3.9100063715395213e-07 by more than 10.0X.`
36. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod2_BLOCK_SIZE_128_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0003401536486123356 is greater than ref error 3.9100063715395213e-07 by more than 10.0X.`
37. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod2_BLOCK_SIZE_256_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0003401536486123356 is greater than ref error 3.9100063715395213e-07 by more than 10.0X.`
38. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_score_mod2_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0003401536486123356 is greater than ref error 3.9100063715395213e-07 by more than 10.0X.`
39. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod2_head_dims0_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 0.0003432532628091305 is greater than ref error 3.7950565240419946e-07 by more than 10.0X.`
40. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_dynamic_score_mask_mod7_xpu_float16` | `failed`
- `AssertionError: Out Compiled error 0.0005424459488396266 is greater than ref error 8.515432279246153e-05 by more than 1.1X.`
41. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_fully_masked_out_rows_xpu` | `failed`
- `AssertionError: Out Compiled error 1.6134144828908867e-05 is greater than ref error 1.50138833154707e-08 by more than 10.0X.`
42. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_captured_buffers_all_dims_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 3.1938262962837545e-05 is greater than ref error 3.0959141599945226e-08 by more than 10.0X.`
43. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod0_BLOCK_SIZE2_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 3.234612694903627e-05 is greater than ref error 2.983891779817198e-08 by more than 10.0X.`
44. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod0_BLOCK_SIZE3_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 3.234612694903627e-05 is greater than ref error 2.983891779817198e-08 by more than 10.0X.`
45. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod0_BLOCK_SIZE_128_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 3.234612694903627e-05 is greater than ref error 2.983891779817198e-08 by more than 10.0X.`
46. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod0_BLOCK_SIZE_256_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 3.234612694903627e-05 is greater than ref error 2.983891779817198e-08 by more than 10.0X.`
47. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_score_mod0_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 3.234612694903627e-05 is greater than ref error 2.983891779817198e-08 by more than 10.0X.`
48. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod0_head_dims0_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 3.2685429287057714e-05 is greater than ref error 2.5822345919188232e-08 by more than 10.0X.`
49. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod0_head_dims1_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 3.301014724820528e-05 is greater than ref error 2.6559310529126298e-08 by more than 10.0X.`
50. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_skip_odd_keys_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 4.343832020544661e-05 is greater than ref error 3.4613310980100195e-08 by more than 10.0X.`
51. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod4_BLOCK_SIZE2_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.1359241903568724e-05 is greater than ref error 3.606854032354876e-08 by more than 10.0X.`
52. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod4_BLOCK_SIZE3_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.1359241903568724e-05 is greater than ref error 3.606854032354876e-08 by more than 10.0X.`
53. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod4_BLOCK_SIZE_128_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.1359241903568724e-05 is greater than ref error 3.606854032354876e-08 by more than 10.0X.`
54. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod4_BLOCK_SIZE_256_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.1359241903568724e-05 is greater than ref error 3.606854032354876e-08 by more than 10.0X.`
55. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_score_mod4_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.1359241903568724e-05 is greater than ref error 3.606854032354876e-08 by more than 10.0X.`
56. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod3_BLOCK_SIZE2_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.1647379950847174e-05 is greater than ref error 3.579670857575287e-08 by more than 10.0X.`
57. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod3_BLOCK_SIZE3_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.1647379950847174e-05 is greater than ref error 3.579670857575287e-08 by more than 10.0X.`
58. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod3_BLOCK_SIZE_128_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.1647379950847174e-05 is greater than ref error 3.579670857575287e-08 by more than 10.0X.`
59. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod3_BLOCK_SIZE_256_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.1647379950847174e-05 is greater than ref error 3.579670857575287e-08 by more than 10.0X.`
60. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_score_mod3_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.1647379950847174e-05 is greater than ref error 3.579670857575287e-08 by more than 10.0X.`
61. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_njt_causal_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.1647379950847174e-05 is greater than ref error 3.579670857575287e-08 by more than 10.0X.`
62. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod4_head_dims0_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.191081641269797e-05 is greater than ref error 3.46471024552142e-08 by more than 10.0X.`
63. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod3_head_dims0_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.233751075635929e-05 is greater than ref error 3.4470122724679894e-08 by more than 10.0X.`
64. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod3_head_dims1_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.248107600049462e-05 is greater than ref error 3.091190081628475e-08 by more than 10.0X.`
65. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod4_head_dims1_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 5.249654540782491e-05 is greater than ref error 3.093247732794174e-08 by more than 10.0X.`
66. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod7_head_dims0_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 9.784255986793805e-05 is greater than ref error 1.9018839104276238e-07 by more than 10.0X.`
67. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod7_BLOCK_SIZE2_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 9.847123440365164e-05 is greater than ref error 1.8829214275010558e-07 by more than 10.0X.`
68. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod7_BLOCK_SIZE3_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 9.847123440365164e-05 is greater than ref error 1.8829214275010558e-07 by more than 10.0X.`
69. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod7_BLOCK_SIZE_128_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 9.847123440365164e-05 is greater than ref error 1.8829214275010558e-07 by more than 10.0X.`
70. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_different_block_size_score_mod7_BLOCK_SIZE_256_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 9.847123440365164e-05 is greater than ref error 1.8829214275010558e-07 by more than 10.0X.`
71. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_builtin_score_mods_score_mod7_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 9.847123440365164e-05 is greater than ref error 1.8829214275010558e-07 by more than 10.0X.`
72. `stock pytorch` | `test/inductor/test_flex_attention.py::TestFlexAttentionXPU::test_non_equal_head_dims_score_mod7_head_dims1_xpu_float32` | `failed`
- `AssertionError: Out Compiled error 9.89134489737822e-05 is greater than ref error 1.816370902440879e-07 by more than 10.0X.`
73. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_absolute_2d_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
74. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_absolute_2d_bias_batch2_head4_seq_len277_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
75. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_absolute_2d_bias_batch2_head4_seq_len37_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
76. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_batch_head_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
77. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_batch_head_bias_batch2_head4_seq_len277_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
78. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_batch_head_bias_batch2_head4_seq_len37_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
79. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_distinct_biases_batch2_head4_seq_len256_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
80. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_distinct_biases_batch2_head4_seq_len277_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
81. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_distinct_biases_batch2_head4_seq_len37_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
82. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_flipped_indexed_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
83. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_flipped_indexed_bias_batch2_head4_seq_len277_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
84. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_flipped_indexed_bias_batch2_head4_seq_len37_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
85. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_global_tokens_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
86. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_global_tokens_bias_batch2_head4_seq_len277_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
87. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_global_tokens_bias_batch2_head4_seq_len37_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
88. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_head_specific_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
89. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_head_specific_bias_batch2_head4_seq_len277_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
90. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_head_specific_bias_batch2_head4_seq_len37_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
91. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_head_specific_gate_batch2_head4_seq_len256_headdim16_dtypefloat32_mode_default_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
92. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_head_specific_gate_batch2_head4_seq_len256_headdim16_dtypefloat32_mode_max-autotune-no-cudagraphs_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
93. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_head_specific_gate_batch2_head4_seq_len277_headdim16_dtypefloat32_mode_default_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
94. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_head_specific_gate_batch2_head4_seq_len277_headdim16_dtypefloat32_mode_max-autotune-no-cudagraphs_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
95. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_head_specific_gate_batch2_head4_seq_len37_headdim16_dtypefloat32_mode_default_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
96. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_head_specific_gate_batch2_head4_seq_len37_headdim16_dtypefloat32_mode_max-autotune-no-cudagraphs_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
97. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_indirect_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
98. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_indirect_bias_batch2_head4_seq_len277_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
99. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_indirect_bias_batch2_head4_seq_len37_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
100. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_local_window_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
101. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_local_window_bias_batch2_head4_seq_len277_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
102. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_local_window_bias_batch2_head4_seq_len37_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
103. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_multiplicative_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
104. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_multiplicative_bias_batch2_head4_seq_len277_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
105. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_multiplicative_bias_batch2_head4_seq_len37_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
106. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_relative_1d_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_mode_default_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
107. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_relative_1d_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_mode_max-autotune-no-cudagraphs_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
108. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_relative_1d_bias_batch2_head4_seq_len277_headdim16_dtypefloat32_mode_default_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
109. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_relative_1d_bias_batch2_head4_seq_len277_headdim16_dtypefloat32_mode_max-autotune-no-cudagraphs_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
110. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_relative_1d_bias_batch2_head4_seq_len37_headdim16_dtypefloat32_mode_default_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
111. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_relative_1d_bias_batch2_head4_seq_len37_headdim16_dtypefloat32_mode_max-autotune-no-cudagraphs_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
112. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_relative_1d_bias_only_grad_batch2_head4_seq_len256_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
113. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_relative_1d_bias_only_grad_batch2_head4_seq_len277_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
114. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_relative_1d_bias_only_grad_batch2_head4_seq_len37_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
115. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_weird_bias_batch2_head4_seq_len256_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
116. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_weird_bias_batch2_head4_seq_len277_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`
117. `stock pytorch` | `test/inductor/test_flex_attention.py::TestLearnableBiasesXPU::test_weird_bias_batch2_head4_seq_len37_headdim16_dtypefloat32_xpu` | `skipped`
- `Compiled FlexAttention is less accurate than eager in fp32`

## Reproducer / Environment Setup

```bash
docker run -it -e TZ=Asia/Shanghai --device=/dev/mem --device=/dev/dri --group-add video --privileged -v $(realpath ${HOME}):/home/jenkins --shm-size=8g intelgpu/ubuntu-26.04-rolling:26.18 bash

apt update
apt upgrade -y

# Expected GPU runtime package versions after upgrade:
dpkg -l | grep -iE 'intel-opencl-icd|libze|libigc|intel-igc|ocloc'
# intel-ocloc 26.22.38646.7-1~26.04~ppa1
# intel-opencl-icd 26.22.38646.7-1~26.04~ppa1
# libigc2 2.36.5-1~26.04
# libze-dev 1.28.6-1~26.04~ppa1
# libze-intel-gpu1 26.22.38646.7-1~26.04~ppa1
# libze1 1.28.6-1~26.04~ppa1

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv myvenv --python 3.10 --clear
source myvenv/bin/activate
uv pip install pip wheel "setuptools<81"
uv pip install torch==2.13.0 torchaudio torchvision --index-url https://download.pytorch.org/whl/xpu

pytorch_commit="$(python -c 'import torch; print(torch.version.git_version)')"
git clone https://github.com/pytorch/pytorch
cd pytorch
git checkout ${pytorch_commit}

uv pip install -r .ci/docker/requirements-ci.txt
uv pip install -U typing_extensions
uv pip install pytest pytest-timeout pytest-xdist pytest-rerunfailures

python test/inductor/test_flex_attention.py -k 'xpu'
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.