[XPU][CI] Random AOTAutograd cache file conflict when tests run in parallel
- Dominant language
- Python
- Stars
- 113
- Forks
- 128
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 112
Description
## Summary
Parallel XPU unit-test runs can randomly fail in `TestAOTAutogradWithCache` because the AOTAutograd local cache hits a file/directory race under `/tmp/torchinductor_root/aotautograd`.
This looks like a cache-concurrency issue rather than a model/test correctness failure: the failure is in cache lookup/save paths, not in the tested operator result.
## Environment
- Repository: `intel/torch-xpu-ops`
- Test file: `test/xpu/functorch/test_aotdispatch_xpu.py`
- Python: 3.10.20
- Parallel runner: pytest-xdist workers observed as `gw1`, `gw6`
- Device: XPU
- Cache root involved: `/tmp/torchinductor_root/aotautograd`
## Failed Tests Observed
Cases:
op_ut,third_party.torch-xpu-ops.test.xpu.functorch.test_aotdispatch_xpu.TestAOTAutogradWithCache,test_input_metadata_mutation_aliases
op_ut,third_party.torch-xpu-ops.test.xpu.functorch.test_aotdispatch_xpu.TestAOTAutogradWithCache,test_input_mutation_simple
op_ut,third_party.torch-xpu-ops.test.xpu.functorch.test_aotdispatch_xpu.TestAOTAutogradWithCache,test_view_and_inplace_view
## Error 1: save path loses its temp file before rename
```text
FileNotFoundError: [Errno 2] No such file or directory:
'/tmp/torchinductor_root/aotautograd/ahbntcmukg7wvan4f3tlvm2a2eziyscbykajslcxsshhlecife32/.396194.140128790849344.tmp'
->
'/tmp/torchinductor_root/aotautograd/ahbntcmukg7wvan4f3tlvm2a2eziyscbykajslcxsshhlecife32/x6hp2ydrhpkbvkkez436t5wezm3wjx4yepc2hjc2jy5vsooiu5q'
W0812 13:32:32.134000 396194 torch/_functorch/_aot_autograd/autograd_cache.py:1412]
AOTAutograd cache unable to serialize compiled graph: [Errno 2] No such file or directory: ...
```
Relevant stack:
```text
torch/_functorch/_aot_autograd/graph_compile.py:2712 in try_save_cache_entry
AOTAutogradCache.save(...)
torch/_functorch/_aot_autograd/autograd_cache.py:1437 in save
AOTAutogradCache._write_to_local_cache(key, content)
torch/_functorch/_aot_autograd/autograd_cache.py:1355 in _write_to_local_cache
write_atomic(path, content)
torch/_inductor/codecache.py:591 in write_atomic
tmp_path.rename(target=path)
```
## Error 2: lookup path sees a missing cache subdir
```text
torch._dynamo.exc.BackendCompilerFailed: backend='dynamo_compiler' raised:
FileNotFoundError: [Errno 2] No such file or directory:
'/tmp/torchinductor_root/aotautograd/aaprgy4cw7hneencncw5xtu4hpotw5tw7f4w7rkzrzibyzck4vzz'
```
Relevant stack:
```text
torch/_functorch/aot_autograd.py:1219 in aot_module_simplified
compiled_fn, aot_config = AOTAutogradCache.try_load(...)
torch/_functorch/_aot_autograd/autograd_cache.py:1085 in try_load
AOTAutogradCache._lookup(...)
torch/_functorch/_aot_autograd/autograd_cache.py:1306 in _lookup
AOTAutogradCache.find_guarded_entry(...)
torch/_inductor/codecache.py:1916 in find_guarded_entry
for candidate, content, in_local in cls.iterate_over_candidates(...)
torch/_inductor/codecache.py:1852 in iterate_over_candidates
for path in sorted(os.listdir(subdir)):
```
## Light Root Cause Analysis
The failure is consistent with a race in the local AOTAutograd/Inductor cache during parallel test execution:
1. All xdist workers use the shared local cache root under `/tmp/torchinductor_root/aotautograd`.
2. `AOTAutogradCache._write_to_local_cache()` creates a key-specific subdirectory and writes the serialized entry via `write_atomic(path, content)`.
3. `write_atomic()` creates a temp file named from `os.getpid()` and `threading.get_ident()`:
```python
tmp_path = path.parent / f".{os.getpid()}.{threading.get_ident()}.tmp"
```
4. That temp filename is not content-specific or operation-specific. Multiple cache writes from the same process/thread into the same key directory can target the same temp path.
5. On the lookup side, `iterate_over_candidates()` checks `os.path.exists(subdir)` and then calls `os.listdir(subdir)`. If another worker/process removes or replaces that cache subdir between the exists check and listdir, lookup raises `FileNotFoundError`.
6. These cache errors then escape into the test as `FileNotFoundError` / `BackendCompilerFailed`, even though the cache is an optimization and should ideally be best-effort under concurrent access.
## Expected Behavior
Parallel test runs should not fail because of local cache file races. If a local AOTAutograd cache entry is concurrently written, removed, or unreadable, the cache path should be treated as a miss/bypass and the test should continue by recompiling.
## Possible Fix Direction
- Make `write_atomic()` temp filenames unique per write operation, for example by including a UUID/random suffix or using `tempfile.NamedTemporaryFile(delete=False, dir=path.parent)`.
- Treat `FileNotFoundError` from `os.listdir(subdir)` as a cache miss in `iterate_over_candidates()`.
- Consider isolating local cache directories per pytest-xdist worker in CI, or disabling the shared local AOTAutograd cache for tests that intentionally exercise cache behavior in parallel.
## Reproducer
Run the XPU AOTAutograd cache tests with pytest-xdist enabled so multiple workers share the same `/tmp/torchinductor_root` cache root. The observed failing workers were `gw1` and `gw6`.
```bash
python -m pytest -n auto -v test/xpu/functorch/test_aotdispatch_xpu.py -k 'TestAOTAutogradWithCache'
```
Contributor guide
Assessment
This issue has not been assessed yet.