intel / intel/torch-xpu-ops

Flaky: test_aot_autograd_cache_xpu.py test_cache_hot_load_device_xpu_bfloat16_dynamic_False (autotune_artifacts count non-deterministic; flaky on 2.13 & 2.14)

Open
#4,922 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

hw: BMG random test: ut
Dominant language
Python
Stars
113
Forks
129
Avg merge
5d 9h
Merged PRs (30d)
112

Description

### Description

Two unit-test cases in `test/xpu/dynamo/test_aot_autograd_cache_xpu.py` fail
**intermittently** (random / flaky). Re-running the same selection produces a mix
of all-pass and 1–2 failures, so they are not deterministic regressions.

**The flakiness reproduces on both `2.13` (baseline) and `2.14` (target)** — it is
not introduced by the 2.14 compiler upgrade; it is a pre-existing flaky test.

Both fail on the same assertion at `test_aot_autograd_cache_xpu.py:225`:

```python
self.assertEqual(len(cache_info.autotune_artifacts), autotune_expect)
```

`autotune_expect` is `2`, but the number of `cache_info.autotune_artifacts`
loaded is non-deterministic across runs — observed **3, 4 and 5**.

### Affected Test Cases

```
test/xpu/dynamo/test_aot_autograd_cache_xpu.py::AOTAutogradCacheTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False
test/xpu/dynamo/test_aot_autograd_cache_xpu.py::AOTAutogradCacheBundledTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False
```

### Observed flakiness

Across 11 back-to-back runs of the 2-case selection (on 2.14):

- `2 passed` — 3 runs
- `1 failed, 1 passed` — 7 runs
- `2 failed` — 1 run

Assertion actual values seen: `got 3` (x1), `got 4` (x6), `got 5` (x2) — always
against `Expected 2`. The same intermittent behavior is seen on 2.13.

### Representative failure

```
_ AOTAutogradCacheBundledTests.test_cache_hot_load_device_xpu_bfloat16_dynamic_False _
Traceback (most recent call last):
File ".../test/xpu/dynamo/test_aot_autograd_cache_xpu.py", line 225, in test_cache_hot_load
self.assertEqual(len(cache_info.autotune_artifacts), autotune_expect)
...
AssertionError: Scalars are not equal!

Expected 2 but got 5.
Absolute difference: 3
Relative difference: 1.5
```

### Reproducer

```bash
docker run -it -e TZ=Asia/Shanghai --device=/dev/mem --device=/dev/dri \
--group-add video --privileged -v $(realpath ${HOME}):/home/jenkins \
--shm-size=8g intelgpu/ubuntu-26.04-rolling:26.18 bash

# python env
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv myvenv --python 3.10 --clear
source myvenv/bin/activate
uv pip install pip wheel "setuptools<81"

# torch — pick ONE (both reproduce the flakiness):
# baseline 2.13
uv pip install torch==2.13.0 torchaudio torchvision --index-url https://download.pytorch.org/whl/xpu
# target 2.14
# uv pip install torch==2.14.0.dev20260805+xpu torchaudio==2.11.0.dev20260806+xpu \
# torchvision==0.29.0.dev20260806+xpu triton-xpu==3.8.0+git1e2d42a0 \
# --index-url https://download.pytorch.org/whl/nightly/xpu --pre

# pytorch source at the matching commit
pytorch_commit="$(python -c 'import torch; print(torch.version.git_version)')"
git clone https://github.com/pytorch/pytorch
cd pytorch
git checkout ${pytorch_commit}

# unit-test deps
uv pip install -r .ci/docker/requirements-ci.txt
uv pip install -U typing_extensions
uv pip install pytest pytest-timeout pytest-xdist pytest-rerunfailures

# torch-xpu-ops tests
cd third_party/
rm -rf torch-xpu-ops && git clone https://github.com/intel/torch-xpu-ops
xpu_ops_commit="$(cat xpu.txt)"
cd torch-xpu-ops
git checkout $xpu_ops_commit
cd test/xpu

# run the two cases; re-run several times to reproduce the intermittent failure
for i in $(seq 1 10); do
python -m pytest -v \
"test_aot_autograd_cache_xpu.py::AOTAutogradCacheTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False" \
"test_aot_autograd_cache_xpu.py::AOTAutogradCacheBundledTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False"
done
```

### Environment

- Reproduces on both `2.13` (baseline) and `2.14` (target) compiler stacks
- Python 3.10, pytest 7.3.2
- PyTorch with `third_party/torch-xpu-ops`
- Device: Intel XPU
- Platform: Linux (`intelgpu/ubuntu-26.04-rolling:26.18`)

### Cases

op_ut,test.xpu.dynamo.test_aot_autograd_cache_xpu.AOTAutogradCacheTests,test_cache_hot_load_device_xpu_bfloat16_dynamic_False
op_ut,test.xpu.dynamo.test_aot_autograd_cache_xpu.AOTAutogradCacheBundledTests,test_cache_hot_load_device_xpu_bfloat16_dynamic_False

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the two named pytest cases repeatedly using the provided XPU environment and reproducer, then read test/xpu/dynamo/test_aot_autograd_cache_xpu.py around line 225 and the cache hot-load setup. Done means both cases pass reliably across repeated runs and the autotune_artifacts count is deterministic with the expected value of 2.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.