Windows: grammar-cache creation can hang when cache writes are denied
- Dominant language
- Python
- Stars
- 14k
- Forks
- 904
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
YAPF 0.43.0 can hang while importing on Windows when its grammar cache directory exists and appears writable to `os.access()`, but creation of the temporary cache file is denied (for example, by a restricted-token sandbox).
Even `yapf --version` hangs before argument processing. The process consumes CPU in Python's temporary-file name retry loop.
This is distinct from #1263: closing/flushing the temporary file before `os.rename()` fixes the open-file rename/cache-persistence problem, but not a denial during the initial `NamedTemporaryFile` creation.
### Environment
- Windows
- YAPF 0.43.0
- Python 3.11.8 from Chromium depot_tools
- Restricted-token filesystem sandbox
YAPF officially supports Python versions where Windows `tempfile.TMP_MAX` can be 2,147,483,647. CPython 3.13 bounds this retry to 20 attempts, but Python 3.11 and 3.12 retain the effectively unbounded behavior.
### Reproduction
Run YAPF in a Windows restricted-token sandbox where the platformdirs cache path under `%LOCALAPPDATA%`:
1. Exists.
2. Is reported writable by `os.access(path, os.W_OK)`.
3. Rejects file creation with `PermissionError` for the sandboxed process.
Then run:
```powershell
python -m yapf --version
```
The same behavior occurs when running YAPF as a directory. No source file is required.
A faulthandler dump from the stuck process ends in:
```text
File ...\Lib\tempfile.py, line 263, in _mkstemp_inner
File ...\Lib\tempfile.py, line 580, in NamedTemporaryFile
File ...\pgen2\grammar.py, line 107, in dump
File ...\pgen2\driver.py, line 242, in load_grammar
File ...\_ylib2to3\pygram.py, line 29, in
```
### Expected behavior
Grammar caching is optional, so a cache write failure should be caught by `load_grammar()` and YAPF should continue uncached.
### Actual behavior
`grammar.Grammar.dump()` calls `tempfile.NamedTemporaryFile()`. On Python 3.11/3.12 for Windows, `tempfile._mkstemp_inner()` treats `PermissionError` as a possible name collision when the directory exists and `os.access()` reports it writable. It retries up to Windows' `TMP_MAX` (2,147,483,647), so control effectively never returns to YAPF's `except OSError` fallback.
There is a second contributing issue in 0.43.0: `Grammar.dump()` calls `os.rename()` while the `NamedTemporaryFile` is still open. On Windows that rename fails, the final cache is never created, and randomly named pickle files are left behind. Consequently YAPF attempts cache generation again on every new process. This appears to be the issue partially addressed by #1263.
### Verified workaround
Pointing platformdirs' Windows override at an existing regular file makes cache-directory creation fail immediately, allowing YAPF's existing `OSError` fallback to continue uncached:
```powershell
$env:WIN_PD_OVERRIDE_LOCAL_APPDATA = (Resolve-Path .\LICENSE).Path
python -m yapf --version
```
### Suggested fix
Please make denied grammar-cache writes reliably fail once and continue uncached, for example by one or more of:
- providing a supported environment variable or option to disable the grammar cache;
- avoiding `NamedTemporaryFile`'s internal Windows retry loop for this optional cache write;
- bounding the cache-write attempt in YAPF itself;
- merging/adapting #1263 so the temporary file is closed before replacement.
Related cache changes/issues: #1243, #1263.
Contributor guide
Research direction
Start with pgen2/grammar.py, where Grammar.dump() creates the temporary cache file, and pgen2/driver.py, where load_grammar() handles cache failures; inspect the import path through _ylib2to3/pygram.py. Reproduce with `python -m yapf --version` in the described Windows restricted-token sandbox. Done means denied optional cache writes return promptly and YAPF continues uncached, with the existing cache behavior preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100