pytest-dev / pytest-dev/pyfakefs
fake_filesystem: RuntimeError "dictionary changed size during iteration" under concurrent mkdir
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 750
- Forks
- 99
- Avg merge
- 19h 6m
- Merged PRs (30d)
- 6
Description
[This bug was found, isolated, fixed, and reported by Claude Opus 4.7, with some minor edits by me.]
Describe the bug
Under concurrent faked mkdir calls that share a parent directory, pyfakefs's internal _directory_content listcomp can iterate a directory-contents dict while another thread is inserting into it. On interpreters whose dict implementation detects concurrent mutation (PyPy; CPython in some configurations) this raises RuntimeError: dictionary changed size during iteration. This is distinct from #1317 (which was about use_original thread-locality) — fixing that bug left this one exposed because the regression test for #1317 exercises concurrent faked mkdir calls.
Stack trace (from PyPy 3.10.16, arm64, running pyfakefs.tests.fake_filesystem_unittest_test.UseOriginalThreadSafetyTest as originally authored in #1318, before the test was restructured to sidestep this bug):
File ".../pyfakefs/fake_filesystem.py", line 1523, in <listcomp>
matching_content = [
RuntimeError: dictionary changed size during iteration
Full trace truncated for brevity; the full trace shows Path.mkdir(parents=True) → _directory_content → the listcomp over directory.contents.items() racing with concurrent .contents[name] = new_child inserts from another thread.
How To Reproduce
The minimal repro requires two threads doing faked mkdir into the same parent dir. With the use_original thread-local fix in place, the following fails reliably on PyPy and may fail intermittently on CPython:
import asyncio
import threading
from pathlib import Path
from pyfakefs.fake_filesystem_unittest import Patcher
def _write(p):
p.parent.mkdir(parents=True, exist_ok=True)
with p.open("w") as cf:
cf.write("x")
async def _dispatch(p):
await asyncio.to_thread(_write, p)
async def many(root, n):
await asyncio.gather(
*[_dispatch(Path(f"{root}/shared/w{i}/file")) for i in range(n)]
)
with Patcher() as p:
p.fs.create_dir("/fake-env")
asyncio.run(many("/fake-env", 64))
All 64 workers share /fake-env/shared/ as a parent directory, racing to insert their own w{i} child. On PyPy this reliably trips the listcomp race.
Your environment
Darwin arm64 (GitHub Actions macOS runner)
PyPy 3.10.16 [arm64]
pyfakefs 6.3.dev0 (commit 10a5538, thread-local-use-original branch)
Not reliably reproducible on CPython 3.12 on linux but is likely latent there (dict mutation during iteration is undefined; CPython can also raise RuntimeError under load).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in fake_filesystem.py around _directory_content at line 1523, then inspect UseOriginalThreadSafetyTest and its concurrent mkdir setup. Run the provided shared-parent reproduction on PyPy and CPython. Done means concurrent faked mkdir calls complete without the dictionary-mutation RuntimeError and regression coverage protects the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100