google / google/yapf

Windows: grammar-cache creation can hang when cache writes are denied

Open
#1,311 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
14k
Forks
904
PR merge metrics
No merged PRs in 30d

Description

### Summary

YAPF 0.43.0 can hang while importing on Windows when its grammar cache directory exists and appears writable to `os.access()`, but creation of the temporary cache file is denied (for example, by a restricted-token sandbox).

Even `yapf --version` hangs before argument processing. The process consumes CPU in Python's temporary-file name retry loop.

This is distinct from #1263: closing/flushing the temporary file before `os.rename()` fixes the open-file rename/cache-persistence problem, but not a denial during the initial `NamedTemporaryFile` creation.

### Environment

- Windows
- YAPF 0.43.0
- Python 3.11.8 from Chromium depot_tools
- Restricted-token filesystem sandbox

YAPF officially supports Python versions where Windows `tempfile.TMP_MAX` can be 2,147,483,647. CPython 3.13 bounds this retry to 20 attempts, but Python 3.11 and 3.12 retain the effectively unbounded behavior.

### Reproduction

Run YAPF in a Windows restricted-token sandbox where the platformdirs cache path under `%LOCALAPPDATA%`:

1. Exists.
2. Is reported writable by `os.access(path, os.W_OK)`.
3. Rejects file creation with `PermissionError` for the sandboxed process.

Then run:

```powershell
python -m yapf --version
```

The same behavior occurs when running YAPF as a directory. No source file is required.

A faulthandler dump from the stuck process ends in:

```text
File ...\Lib\tempfile.py, line 263, in _mkstemp_inner
File ...\Lib\tempfile.py, line 580, in NamedTemporaryFile
File ...\pgen2\grammar.py, line 107, in dump
File ...\pgen2\driver.py, line 242, in load_grammar
File ...\_ylib2to3\pygram.py, line 29, in
```

### Expected behavior

Grammar caching is optional, so a cache write failure should be caught by `load_grammar()` and YAPF should continue uncached.

### Actual behavior

`grammar.Grammar.dump()` calls `tempfile.NamedTemporaryFile()`. On Python 3.11/3.12 for Windows, `tempfile._mkstemp_inner()` treats `PermissionError` as a possible name collision when the directory exists and `os.access()` reports it writable. It retries up to Windows' `TMP_MAX` (2,147,483,647), so control effectively never returns to YAPF's `except OSError` fallback.

There is a second contributing issue in 0.43.0: `Grammar.dump()` calls `os.rename()` while the `NamedTemporaryFile` is still open. On Windows that rename fails, the final cache is never created, and randomly named pickle files are left behind. Consequently YAPF attempts cache generation again on every new process. This appears to be the issue partially addressed by #1263.

### Verified workaround

Pointing platformdirs' Windows override at an existing regular file makes cache-directory creation fail immediately, allowing YAPF's existing `OSError` fallback to continue uncached:

```powershell
$env:WIN_PD_OVERRIDE_LOCAL_APPDATA = (Resolve-Path .\LICENSE).Path
python -m yapf --version
```

### Suggested fix

Please make denied grammar-cache writes reliably fail once and continue uncached, for example by one or more of:

- providing a supported environment variable or option to disable the grammar cache;
- avoiding `NamedTemporaryFile`'s internal Windows retry loop for this optional cache write;
- bounding the cache-write attempt in YAPF itself;
- merging/adapting #1263 so the temporary file is closed before replacement.

Related cache changes/issues: #1243, #1263.

Contributor guide

Open the contributing guide

Research direction

Start with pgen2/grammar.py, where Grammar.dump() creates the temporary cache file, and pgen2/driver.py, where load_grammar() handles cache failures; inspect the import path through _ylib2to3/pygram.py. Reproduce with `python -m yapf --version` in the described Windows restricted-token sandbox. Done means denied optional cache writes return promptly and YAPF continues uncached, with the existing cache behavior preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.