lance-format / lance-format/lance

bug: Windows CI flake - pytest-xdist workers race on the shared torchinductor cache

Open Beginner friendly
#9,055 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

The windows job fails intermittently on main and on PRs with a PermissionError on a file under torchinductor_runneradmin, in whichever torch-based test happens to lose the race. It is not tied to a particular test or to the change under review.

Occurrences:

torch._inductor.exc.InductorError: PermissionError: [Errno 13] Permission denied:
'C:\Users\RUNNER~1\AppData\Local\Temp\torchinductor_runneradmin\5h\c5h7jepkg4o4vxj5qik5e3dax3tsrhzgga3lujk3z74ctjdxcg3i.py'

The first two hit the identical cache path. That filename is a content hash of the generated kernel, so both runs were compiling the same kernel, and every failure so far names a different worker. torchinductor_runneradmin is a single per-user directory shared by all pytest-xdist workers, and on Windows a file held open by one process cannot be replaced by another, so two workers compiling the same kernel collide instead of harmlessly rewriting it.

A per-worker cache directory should remove the collision, e.g. deriving TORCHINDUCTOR_CACHE_DIR from PYTEST_XDIST_WORKER in the Windows test step.

Related: #8789 listed a different Windows temp race (test_build_ivf_flat::case_3) and is closed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Windows test step in the GitHub Actions configuration and review the listed failing runs to confirm the shared torchinductor cache behavior. Configure a cache directory derived from PYTEST_XDIST_WORKER, then rerun the Windows pytest-xdist job and verify that concurrent workers no longer collide.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python, pytorch
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.