Flaky: test_aot_autograd_cache_xpu.py test_cache_hot_load_device_xpu_bfloat16_dynamic_False (autotune_artifacts count non-deterministic; flaky on 2.13 & 2.14)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 113
- Forks
- 129
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
### Description
Two unit-test cases in `test/xpu/dynamo/test_aot_autograd_cache_xpu.py` fail
**intermittently** (random / flaky). Re-running the same selection produces a mix
of all-pass and 1–2 failures, so they are not deterministic regressions.
**The flakiness reproduces on both `2.13` (baseline) and `2.14` (target)** — it is
not introduced by the 2.14 compiler upgrade; it is a pre-existing flaky test.
Both fail on the same assertion at `test_aot_autograd_cache_xpu.py:225`:
```python
self.assertEqual(len(cache_info.autotune_artifacts), autotune_expect)
```
`autotune_expect` is `2`, but the number of `cache_info.autotune_artifacts`
loaded is non-deterministic across runs — observed **3, 4 and 5**.
### Affected Test Cases
```
test/xpu/dynamo/test_aot_autograd_cache_xpu.py::AOTAutogradCacheTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False
test/xpu/dynamo/test_aot_autograd_cache_xpu.py::AOTAutogradCacheBundledTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False
```
### Observed flakiness
Across 11 back-to-back runs of the 2-case selection (on 2.14):
- `2 passed` — 3 runs
- `1 failed, 1 passed` — 7 runs
- `2 failed` — 1 run
Assertion actual values seen: `got 3` (x1), `got 4` (x6), `got 5` (x2) — always
against `Expected 2`. The same intermittent behavior is seen on 2.13.
### Representative failure
```
_ AOTAutogradCacheBundledTests.test_cache_hot_load_device_xpu_bfloat16_dynamic_False _
Traceback (most recent call last):
File ".../test/xpu/dynamo/test_aot_autograd_cache_xpu.py", line 225, in test_cache_hot_load
self.assertEqual(len(cache_info.autotune_artifacts), autotune_expect)
...
AssertionError: Scalars are not equal!
Expected 2 but got 5.
Absolute difference: 3
Relative difference: 1.5
```
### Reproducer
```bash
docker run -it -e TZ=Asia/Shanghai --device=/dev/mem --device=/dev/dri \
--group-add video --privileged -v $(realpath ${HOME}):/home/jenkins \
--shm-size=8g intelgpu/ubuntu-26.04-rolling:26.18 bash
# python env
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv myvenv --python 3.10 --clear
source myvenv/bin/activate
uv pip install pip wheel "setuptools<81"
# torch — pick ONE (both reproduce the flakiness):
# baseline 2.13
uv pip install torch==2.13.0 torchaudio torchvision --index-url https://download.pytorch.org/whl/xpu
# target 2.14
# uv pip install torch==2.14.0.dev20260805+xpu torchaudio==2.11.0.dev20260806+xpu \
# torchvision==0.29.0.dev20260806+xpu triton-xpu==3.8.0+git1e2d42a0 \
# --index-url https://download.pytorch.org/whl/nightly/xpu --pre
# pytorch source at the matching commit
pytorch_commit="$(python -c 'import torch; print(torch.version.git_version)')"
git clone https://github.com/pytorch/pytorch
cd pytorch
git checkout ${pytorch_commit}
# unit-test deps
uv pip install -r .ci/docker/requirements-ci.txt
uv pip install -U typing_extensions
uv pip install pytest pytest-timeout pytest-xdist pytest-rerunfailures
# torch-xpu-ops tests
cd third_party/
rm -rf torch-xpu-ops && git clone https://github.com/intel/torch-xpu-ops
xpu_ops_commit="$(cat xpu.txt)"
cd torch-xpu-ops
git checkout $xpu_ops_commit
cd test/xpu
# run the two cases; re-run several times to reproduce the intermittent failure
for i in $(seq 1 10); do
python -m pytest -v \
"test_aot_autograd_cache_xpu.py::AOTAutogradCacheTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False" \
"test_aot_autograd_cache_xpu.py::AOTAutogradCacheBundledTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False"
done
```
### Environment
- Reproduces on both `2.13` (baseline) and `2.14` (target) compiler stacks
- Python 3.10, pytest 7.3.2
- PyTorch with `third_party/torch-xpu-ops`
- Device: Intel XPU
- Platform: Linux (`intelgpu/ubuntu-26.04-rolling:26.18`)
### Cases
op_ut,test.xpu.dynamo.test_aot_autograd_cache_xpu.AOTAutogradCacheTests,test_cache_hot_load_device_xpu_bfloat16_dynamic_False
op_ut,test.xpu.dynamo.test_aot_autograd_cache_xpu.AOTAutogradCacheBundledTests,test_cache_hot_load_device_xpu_bfloat16_dynamic_False
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run the two named pytest cases repeatedly using the provided XPU environment and reproducer, then read test/xpu/dynamo/test_aot_autograd_cache_xpu.py around line 225 and the cache hot-load setup. Done means both cases pass reliably across repeated runs and the autotune_artifacts count is deterministic with the expected value of 2.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100