pytest-dev / pytest-dev/pyfakefs

fake_filesystem: RuntimeError "dictionary changed size during iteration" under concurrent mkdir

Open
#1,319 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
750
Forks
99
Avg merge
19h 6m
Merged PRs (30d)
6

Description

[This bug was found, isolated, fixed, and reported by Claude Opus 4.7, with some minor edits by me.]

Describe the bug

Under concurrent faked mkdir calls that share a parent directory, pyfakefs's internal _directory_content listcomp can iterate a directory-contents dict while another thread is inserting into it. On interpreters whose dict implementation detects concurrent mutation (PyPy; CPython in some configurations) this raises RuntimeError: dictionary changed size during iteration. This is distinct from #1317 (which was about use_original thread-locality) — fixing that bug left this one exposed because the regression test for #1317 exercises concurrent faked mkdir calls.

Stack trace (from PyPy 3.10.16, arm64, running pyfakefs.tests.fake_filesystem_unittest_test.UseOriginalThreadSafetyTest as originally authored in #1318, before the test was restructured to sidestep this bug):

File ".../pyfakefs/fake_filesystem.py", line 1523, in <listcomp>
    matching_content = [
RuntimeError: dictionary changed size during iteration

Full trace truncated for brevity; the full trace shows Path.mkdir(parents=True)_directory_content → the listcomp over directory.contents.items() racing with concurrent .contents[name] = new_child inserts from another thread.

How To Reproduce

The minimal repro requires two threads doing faked mkdir into the same parent dir. With the use_original thread-local fix in place, the following fails reliably on PyPy and may fail intermittently on CPython:

import asyncio
import threading
from pathlib import Path
from pyfakefs.fake_filesystem_unittest import Patcher

def _write(p):
    p.parent.mkdir(parents=True, exist_ok=True)
    with p.open("w") as cf:
        cf.write("x")

async def _dispatch(p):
    await asyncio.to_thread(_write, p)

async def many(root, n):
    await asyncio.gather(
        *[_dispatch(Path(f"{root}/shared/w{i}/file")) for i in range(n)]
    )

with Patcher() as p:
    p.fs.create_dir("/fake-env")
    asyncio.run(many("/fake-env", 64))

All 64 workers share /fake-env/shared/ as a parent directory, racing to insert their own w{i} child. On PyPy this reliably trips the listcomp race.

Your environment

Observed on:

Darwin arm64 (GitHub Actions macOS runner)
PyPy 3.10.16 [arm64]
pyfakefs 6.3.dev0 (commit 10a5538, thread-local-use-original branch)

Not reliably reproducible on CPython 3.12 on linux but is likely latent there (dict mutation during iteration is undefined; CPython can also raise RuntimeError under load).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in fake_filesystem.py around _directory_content at line 1523, then inspect UseOriginalThreadSafetyTest and its concurrent mkdir setup. Run the provided shared-parent reproduction on PyPy and CPython. Done means concurrent faked mkdir calls complete without the dictionary-mutation RuntimeError and regression coverage protects the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.