Test failure due to (presumed) race condition
Open
- Dominant language
- Python
- Stars
- 59
- Forks
- 32
- PR merge metrics
- No merged PRs in 30d
Description
Sometimes (once in a few months) our CI fails with peculiar error like this
```
FAILED tests/kernel/wave/attention/decode_attention_test.py::testFlashDecoding[MMAType.F32_16x16x16_F16-dyn-SchedulingType.NONE-40x1024x64x64x1024] - AssertionError: Tensor-likes are not close!
Mismatched elements: 0 / 2621440 (0.0%)
Greatest absolute difference: 0.0 at index (0, 0, 0) (up to 0.001 allowed)
Greatest relative difference: 0.0 at index (0, 0, 0) (up to 0.001 allowed)
= 1 failed, 612 passed, 1768 skipped, 1 xfailed, 45 warnings in 709.55s (0:11:49)
```
Which is most likely due to race condition when correct data arrived in the middle of torch doing `assert_close`
Last time observed on mi250 runner.
Contributor guide
Assessment
This issue has not been assessed yet.