lance-format / lance-format/lance
bug: Windows CI flake - pytest-xdist workers race on the shared torchinductor cache
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
The windows job fails intermittently on main and on PRs with a PermissionError on a file under torchinductor_runneradmin, in whichever torch-based test happens to lose the race. It is not tied to a particular test or to the change under review.
Occurrences:
- PR #9054,
test_torch_index_with_nans[Legacy], workergw0- https://github.com/lance-format/lance/actions/runs/34195246750/job/101961424216 main,test_torch_index_with_nans[V3], workergw3- https://github.com/lance-format/lance/actions/runs/33933386340/job/101216448469main,test_index_cast_centroids, workergw4- https://github.com/lance-format/lance/actions/runs/33893404408/job/101090259544
torch._inductor.exc.InductorError: PermissionError: [Errno 13] Permission denied:
'C:\Users\RUNNER~1\AppData\Local\Temp\torchinductor_runneradmin\5h\c5h7jepkg4o4vxj5qik5e3dax3tsrhzgga3lujk3z74ctjdxcg3i.py'
The first two hit the identical cache path. That filename is a content hash of the generated kernel, so both runs were compiling the same kernel, and every failure so far names a different worker. torchinductor_runneradmin is a single per-user directory shared by all pytest-xdist workers, and on Windows a file held open by one process cannot be replaced by another, so two workers compiling the same kernel collide instead of harmlessly rewriting it.
A per-worker cache directory should remove the collision, e.g. deriving TORCHINDUCTOR_CACHE_DIR from PYTEST_XDIST_WORKER in the Windows test step.
Related: #8789 listed a different Windows temp race (test_build_ivf_flat::case_3) and is closed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Windows test step in the GitHub Actions configuration and review the listed failing runs to confirm the shared torchinductor cache behavior. Configure a cache directory derived from PYTEST_XDIST_WORKER, then rerun the Windows pytest-xdist job and verify that concurrent workers no longer collide.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python, pytorch
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 76/100