Intermittent AOTAutograd cache hot-load XPU tests report extra autotune artifacts
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
## Summary
Two XPU AOTAutograd cache hot-load tests failed intermittently when repeated 16 times
## Repeated test results pass rate
dynamo/test_aot_autograd_cache_xpu.py::AOTAutogradCacheBundledTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False 5 / 16
dynamo/test_aot_autograd_cache_xpu.py::AOTAutogradCacheTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False 9 / 16
Cases:
op_ut,third_party.torch-xpu-ops.test.xpu.dynamo.test_aot_autograd_cache_xpu.AOTAutogradCacheBundledTests,test_cache_hot_load_device_xpu_bfloat16_dynamic_False
op_ut,third_party.torch-xpu-ops.test.xpu.dynamo.test_aot_autograd_cache_xpu.AOTAutogradCacheTests,test_cache_hot_load_device_xpu_bfloat16_dynamic_False
Device: Intel(R) Arc(TM) Pro B60 Graphics
## Reproducer
```
docker run -it -e TZ=Asia/Shanghai --device=/dev/mem --device=/dev/dri --group-add video --privileged -v $(realpath ${HOME}):/home/jenkins --shm-size=8g intelgpu/ubuntu-26.04-rolling:26.18 bash
# setup python
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv myvenv --python 3.10 --clear
source myvenv/bin/activate
uv pip install pip wheel "setuptools<81"
# install pytorch
export GH_TOKEN=xxx
gh --repo intel/torch-xpu-ops run download 31671453623 -p "Torch-XPU-Wheel-*"
uv pip install Torch-XPU-Wheel-4812-31671453623-1/*.whl
pytorch_commit="$(python -c 'import torch; print(torch.version.git_version)')"
git clone https://github.com/pytorch/pytorch
cd pytorch
git checkout ${pytorch_commit}
# install for unit tests
uv pip install -r .ci/docker/requirements-ci.txt
uv pip install -U typing_extensions
uv pip install pytest pytest-timeout pytest-xdist pytest-rerunfailures
# pytest
cd third_party/
git clone https://github.com/intel/torch-xpu-ops
cd torch-xpu-ops
cd test/xpu
```
## Failure signature
Both tests fail with the same assertion in `test_cache_hot_load`:
```text
File "/workspace/pytorch/third_party/torch-xpu-ops/test/xpu/dynamo/test_aot_autograd_cache_xpu.py", line 225, in test_cache_hot_load
self.assertEqual(len(cache_info.autotune_artifacts), autotune_expect)
AssertionError: Scalars are not equal!
Expected 2 but got 4.
Absolute difference: 2
Relative difference: 1.0
```
Representative failed logs:
```text
/tmp/xpu_repeat_20260814_162727/logs/case2_iter1_gpu0.log
/tmp/xpu_repeat_20260814_162727/logs/case3_iter5_gpu4.log
```
## Repro commands
```bash
PYTORCH_TEST_WITH_SLOW=1 pytest -q dynamo/test_aot_autograd_cache_xpu.py::AOTAutogradCacheBundledTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False
PYTORCH_TEST_WITH_SLOW=1 pytest -q dynamo/test_aot_autograd_cache_xpu.py::AOTAutogradCacheTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False
```
Contributor guide
Research direction
Start with third_party/torch-xpu-ops/test/xpu/dynamo/test_aot_autograd_cache_xpu.py and run the two named test_cache_hot_load pytest commands repeatedly on XPU. Inspect the assertion around cache_info.autotune_artifacts and the failing logs to determine why four artifacts appear instead of two. Done means both tests pass reliably without extra autotune artifacts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100