Sharing a dict iterator across threads double-DECREFs di_dict under free-threading

Crash report

What happened?

On a free-threaded build, advancing a single, shared dict iterator (iter({...})) from multiple threads double-frees the underlying dict and crashes.

All three dict iterators (dict_keyiterator / dict_valueiterator / dict_itemiterator) route next() through dictiter_iternext_threadsafe (Objects/dictobject.c). Its exhaustion path drops the iterator's owning reference to the dict:

fail:
    di->di_dict = NULL;   /* non-atomic clear */
    Py_DECREF(d);         /* drop the iterator's ONE owning ref to the dict */
    return -1;

and the caller reads that reference with no lock:

static PyObject*
dictiter_iternextkey(PyObject *self)
{
    dictiterobject *di = (dictiterobject *)self;
    PyDictObject *d = di->di_dict;      /* plain read */
    if (d == NULL)
        return NULL;
    PyObject *value;
    if (dictiter_iternext_threadsafe(d, self, &value, NULL) < 0) {
        value = NULL;
    }
    return value;
}

The iterator holds exactly one reference to the dict (di_dict). Two threads calling next() on the same near-exhausted iterator interleave as: both read d = di->di_dict (non-NULL), both reach fail:, and both run Py_DECREF(d). The second Py_DECREF has no matching reference — the dict's refcount underflows / it is freed one owner too early, and a sibling thread still walking d (and the dict freelist / keys object) then touches freed memory.

This fail: di->di_dict = NULL; Py_DECREF(d) pattern predates free-threading (it is correct under the GIL, where only one thread runs the iterator); the lock-free dictiter_iternext_threadsafe wrapper carried it over unguarded.

Reproducer

import threading

NT = 8
ITERS = 200_000

def newit():
    return iter({k: k for k in range(32)})

cell = [newit()]

def worker():
    for _ in range(ITERS):
        it = cell[0]
        try:
            next(it)                 # dictiter_iternextkey -> dictiter_iternext_threadsafe
        except StopIteration:
            cell[0] = newit()        # refill so the fail: (exhaustion) path is hit repeatedly
        except Exception:
            pass

threads = [threading.Thread(target=worker) for _ in range(NT)]
for t in threads: t.start()
for t in threads: t.join()
print("done, no crash")

PYTHON_GIL=0 ./python repro.py on a free-threaded build:

• Debug build → SIGABRT within seconds, ~8/8 runs, with _Py_NegativeRefcount on the dict (Objects/dictobject.c:6159), plus downstream corruption faces as the freed dict/keys object is reused (dictkeys_incref immortal-refcount assert :484; new_dict type assert :978; clear_freelist Objects/object.c:909; validate_refcounts in gc_free_threading.c).
• Release build (-O0, no sanitizer) → SIGSEGV (use-after-free), or occasionally Fatal Python error: PyMutex_Unlock: unlocking mutex that is not locked from the corrupted dict mutex.

So it is neither debug-only nor sanitizer-only. Debug backtrace (the negative-refcount object is the dict d):

#8  _PyObject_AssertFailed (obj=0x...dict...) at Objects/object.c:3278
#9  _Py_NegativeRefcount               at Objects/object.c:275
#12 Py_DECREF                          at ./Include/refcount.h:363
#13 dictiter_iternext_threadsafe (d=0x...dict...) at Objects/dictobject.c:6159    <-- fail: Py_DECREF(d)
#14 dictiter_iternextkey                at Objects/dictobject.c:5791
#15 builtin_next                        at Python/bltinmodule.c:1776

Suggested fix

Consume the reference atomically so exactly one thread performs the DECREF:

fail:
    PyDictObject *old = _Py_atomic_exchange_ptr(&di->di_dict, NULL);
    if (old != NULL) {
        Py_DECREF(old);
    }
    return -1;

and keep the dict alive for the duration of the lock-free walk (the caller uses d and its keys/values across the whole dictiter_iternext_threadsafe body) so a sibling that wins the exchange cannot free it mid-iteration — take a strong reference or the dict's critical section for the walk. One fix covers keys/values/items, since all three route through dictiter_iternext_threadsafe.

Why this is not the documented value-benign iterator race

This function was written to be shared across threads, and a crash is explicitly out of contract:

• The lock-free dict iterator dictiter_iternext_threadsafe was added by gh-112075 / PR #115108 ("Iterating a dict shouldn't require locks"), under the umbrella gh-112075 ("Make dict objects thread-safe in --disable-gil builds"). PR #115108 states it "[handles] races against the dict as well as allowing the iterator to be used from multiple threads simultaneously." It made the value read safe (_Py_TryIncrefCompare / acquire_key_value) but carried the old fail: di->di_dict = NULL; Py_DECREF(d) exhaustion path in unchanged — hence this double-free.
• gh-120496 ("Sequence iterator thread-safety") decided not to fix the fact that concurrent iteration can return duplicate/skipped values, and documented it instead (only the Doc/glossary.rst note, PR #120685, merged; the code-fix PRs were closed). But the contract agreed on that issue is explicit — @colesbury and @eendebakpt: iterating from multiple threads "will not crash the interpreter" (that's the acceptable line; wrong values are OK, crashes are not), and @eendebakpt flagged "the risks of overflows inside the C code."
• gh-148873 reported this iterator's data-race face and was closed as a duplicate of gh-120496 — folding a double-free into the value-benign class. That is the gap this issue closes: the same unsynchronized clear-and-DECREF is not benign — it double-frees the dict and crashes (negative refcount / UAF / SIGSEGV) on plain free-threaded builds, which gh-120496 / gh-124397 put squarely on the not-acceptable side.

(Found by fusil --tsan, a ThreadSanitizer fuzzer; crash confirmed by re-running the reproducer without a sanitizer on a plain free-threaded build. Draft and reproducer by Claude Code, minimized and reviewed by hand.)

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.16.0a0 free-threading build (heads/main:a1d580430c8, Jul 18 2026, 20:23:36) [Clang 21.1.8 (6ubuntu1)]

Comience en Objects/dictobject.c, en dictiter_iternext_threadsafe y sus callers dictiter_iternextkey, dictiter_iternextvalue y dictiter_iternextitem; compare la ruta de agotamiento con las suposiciones sobre iteradores compartidos descritas en el issue. Ejecute el reproducer proporcionado en un free-threaded build y, a continuación, verifique que el agotamiento repetido en las rutas de keys, values e items ya no produzca refcount failures ni use-after-free.