intel / intel/torch-xpu-ops

Intermittent AOTAutograd cache hot-load XPU tests report extra autotune artifacts

Open
#4,964 0 comments 0 reactions 0 assignees View on GitHub
hw: BMG random skipped
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

## Summary

Two XPU AOTAutograd cache hot-load tests failed intermittently when repeated 16 times

## Repeated test results pass rate

dynamo/test_aot_autograd_cache_xpu.py::AOTAutogradCacheBundledTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False 5 / 16
dynamo/test_aot_autograd_cache_xpu.py::AOTAutogradCacheTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False 9 / 16

Cases:
op_ut,third_party.torch-xpu-ops.test.xpu.dynamo.test_aot_autograd_cache_xpu.AOTAutogradCacheBundledTests,test_cache_hot_load_device_xpu_bfloat16_dynamic_False
op_ut,third_party.torch-xpu-ops.test.xpu.dynamo.test_aot_autograd_cache_xpu.AOTAutogradCacheTests,test_cache_hot_load_device_xpu_bfloat16_dynamic_False

Device: Intel(R) Arc(TM) Pro B60 Graphics

## Reproducer
```
docker run -it -e TZ=Asia/Shanghai --device=/dev/mem --device=/dev/dri --group-add video --privileged -v $(realpath ${HOME}):/home/jenkins --shm-size=8g intelgpu/ubuntu-26.04-rolling:26.18 bash

# setup python
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv myvenv --python 3.10 --clear
source myvenv/bin/activate
uv pip install pip wheel "setuptools<81"

# install pytorch
export GH_TOKEN=xxx
gh --repo intel/torch-xpu-ops run download 31671453623 -p "Torch-XPU-Wheel-*"
uv pip install Torch-XPU-Wheel-4812-31671453623-1/*.whl
pytorch_commit="$(python -c 'import torch; print(torch.version.git_version)')"
git clone https://github.com/pytorch/pytorch
cd pytorch
git checkout ${pytorch_commit}

# install for unit tests
uv pip install -r .ci/docker/requirements-ci.txt
uv pip install -U typing_extensions
uv pip install pytest pytest-timeout pytest-xdist pytest-rerunfailures

# pytest
cd third_party/
git clone https://github.com/intel/torch-xpu-ops
cd torch-xpu-ops
cd test/xpu

```

## Failure signature

Both tests fail with the same assertion in `test_cache_hot_load`:

```text
File "/workspace/pytorch/third_party/torch-xpu-ops/test/xpu/dynamo/test_aot_autograd_cache_xpu.py", line 225, in test_cache_hot_load
self.assertEqual(len(cache_info.autotune_artifacts), autotune_expect)
AssertionError: Scalars are not equal!

Expected 2 but got 4.
Absolute difference: 2
Relative difference: 1.0
```

Representative failed logs:

```text
/tmp/xpu_repeat_20260814_162727/logs/case2_iter1_gpu0.log
/tmp/xpu_repeat_20260814_162727/logs/case3_iter5_gpu4.log
```

## Repro commands

```bash
PYTORCH_TEST_WITH_SLOW=1 pytest -q dynamo/test_aot_autograd_cache_xpu.py::AOTAutogradCacheBundledTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False
PYTORCH_TEST_WITH_SLOW=1 pytest -q dynamo/test_aot_autograd_cache_xpu.py::AOTAutogradCacheTests::test_cache_hot_load_device_xpu_bfloat16_dynamic_False
```

Contributor guide

Open the contributing guide

Research direction

Start with third_party/torch-xpu-ops/test/xpu/dynamo/test_aot_autograd_cache_xpu.py and run the two named test_cache_hot_load pytest commands repeatedly on XPU. Inspect the assertion around cache_info.autotune_artifacts and the failing logs to determine why four artifacts appear instead of two. Done means both tests pass reliably without extra autotune artifacts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.