`dir()` can raise `RuntimeError: dictionary changed size during iteration` on the free-threaded build

Bug report

Bug description:

I believe there is an issue in the free-threaded build when using dir(). Apologies if this isn't considered an issue. I read https://docs.python.org/3/howto/free-threading-python.html#thread-safety and my understanding of:

Built-in types like dict, list, and set use internal locks to protect against concurrent modifications in ways that behave similarly to the GIL.

is that this is something that could be fixed. Possibly this was missed when implementing https://github.com/python/cpython/pull/114508.

Calling dir() on a class (or on an instance of it) can fail with RuntimeError: dictionary changed size during iteration if another thread concurrently performs a read that lazily stores something in the class __dict__. The most common such read on 3.14+ is the first access to __annotations__ (e.g. through typing.get_type_hints()), which stores __annotations_cache__ on the class.

Same can happen with e.g. copy.copy(), which will set __slotnames__ via copyreg._slotnames().

MRE

import sys
import threading
import typing
from concurrent.futures import ThreadPoolExecutor


def make_class():
    class C:
        x: int

    # Insert many elements in the class dict so that `dir()` spends more time
    # iterating over it (the race also happens without this, just less often):
    for i in range(1000):
        setattr(C, f'attr_{i}', i)
    return C


def main(rounds: int = 100, readers: int = 8) -> None:
    failures = 0
    for _ in range(rounds):
        C = make_class()
        barrier = threading.Barrier(readers + 1)
        errors = []

        def read():
            barrier.wait()
            try:
                dir(C)
            except RuntimeError as e:
                errors.append(e)

        def write():
            barrier.wait()
            # First access to `C.__annotations__` (here, through `get_type_hints()`) stores
            # `__annotations_cache__` in the class `__dict__`.
            typing.get_type_hints(C)

        with ThreadPoolExecutor(max_workers=readers + 1) as executor:
            futures = [executor.submit(read) for _ in range(readers)] + [executor.submit(write)]
            for future in futures:
                future.result()
        failures += len(errors)
        if errors and failures == len(errors):
            print(f'{type(errors[0]).__name__}: {errors[0]}')

    gil = sys._is_gil_enabled()
    print(f'{sys.version.split()[0]} ({"GIL" if gil else "free-threaded"}): {failures}/{rounds * readers} dir() calls failed')


if __name__ == '__main__':
    main()

$ python3.15t mre.py
RuntimeError: dictionary changed size during iteration
3.15.0rc2 (free-threaded): 108/800 dir() calls failed
$ PYTHON_GIL=1 python3.14t mre.py
3.15.0rc2 (GIL): 0/800 dir() calls failed

Analysis

AI analysis pointed me at the following. I'm not knowledgeable to know if this is the actual issue, but can help for initial debugging.

The reader: dir()

type.__dir__() and object.__dir__() collect names with merge_class_dict(), which fetches cls.__dict__ and merges it into a fresh dict with PyDict_Update():

https://github.com/python/cpython/blob/894af95edf7d5a1b0ebdc3699889743c8f41baf1/Objects/typeobject.c#L6965-L6982

cls.__dict__ is a mappingproxy, not a dict, so dict_merge() doesn't take the fast path (which holds the critical sections of both dicts). It takes the generic path instead, which only holds the critical section of the destination dict:

https://github.com/python/cpython/blob/894af95edf7d5a1b0ebdc3699889743c8f41baf1/Objects/dictobject.c#L4309-L4324

The generic path gets the keys with PyMapping_Keys(), which for a non-dict calls .keys() and turns the returned dict_keys view into a list by iterating over it:

https://github.com/python/cpython/blob/894af95edf7d5a1b0ebdc3699889743c8f41baf1/Objects/abstract.c#L2433-L2468

That iteration over the class __dict__ happens without holding its critical section, and the dict iterator checks the size at every step, so any concurrent insertion into the class __dict__ raises RuntimeError.

The writer: a lazy cache stored on the class by a read

The type.__annotations__ getter stores the evaluated annotations as __annotations_cache__ in the class __dict__ on first access:

https://github.com/python/cpython/blob/894af95edf7d5a1b0ebdc3699889743c8f41baf1/Objects/typeobject.c#L2194-L2201

If the class has no __annotate__ function, the type.__annotate__ getter (called by the above) also stores __annotate_func__ = None:

https://github.com/python/cpython/blob/894af95edf7d5a1b0ebdc3699889743c8f41baf1/Objects/typeobject.c#L2086-L2092

Other stdlib operations do the same kind of lazy insertion into a class __dict__, so the issue is not specific to annotations:

• copy.copy(), copy.deepcopy() and pickling of an instance store __slotnames__ on the class, on every version, through _PyType_GetSlotNames() / copyreg._slotnames():

  https://github.com/python/cpython/blob/894af95edf7d5a1b0ebdc3699889743c8f41baf1/Objects/typeobject.c#L7790-L7795

  https://github.com/python/cpython/blob/894af95edf7d5a1b0ebdc3699889743c8f41baf1/Lib/copyreg.py#L155-L158

• The first instantiation of a concrete subclass of a typing.Protocol replaces its __init__():

  https://github.com/python/cpython/blob/894af95edf7d5a1b0ebdc3699889743c8f41baf1/Lib/typing.py#L1925-L1931

Other affected readers

Everything relying on dir() is affected, e.g. inspect.getmembers() and inspect.classify_class_attrs() (which additionally iterate over base.__dict__.items() themselves), or unittest.mock's autospec. dict(vars(cls)) (used e.g. by typing.get_type_hints() for the evaluation locals) goes through the same dict_merge() generic path:

https://github.com/python/cpython/blob/894af95edf7d5a1b0ebdc3699889743c8f41baf1/Lib/typing.py#L2451

Possible fix

In dict_merge() (or in PyMapping_Keys()), unwrap a mappingproxy whose underlying mapping is a dict and use the locked dict-to-dict path. This would fix dir(), dict(vars(cls)), {**vars(cls)} and everything built on them at once. The pure-Python loops over base.__dict__.items() in inspect and enum.Enum.__dir__ would still need to iterate over a copy.

CPython versions tested on:

3.15

Operating systems tested on:

macOS

Linked PRs

• gh-157279

Reproduza a condição de corrida com o MRE free-threaded fornecido e, em seguida, revise gh-157279 antes de fazer alterações. Leia typeobject.c, dictobject.c e abstract.c ao redor dos pontos de entrada vinculados e identifique o local relevante do teste de regressão. A conclusão deve incluir cobertura para leituras concorrentes de dicionários de classe e escritas lazy sem o RuntimeError.