Free-threading: GC untracks immortal objects, violating the always-tracked dict invariant asserted by dict_merge_api (debug abort at shutdown)

Bug report

Summary

On the free-threaded build, the cyclic GC permanently untracks any immortal object it
encounters during collection. Free-threaded dicts, however, are expected to stay GC-tracked:
dict_merge_api() asserts _PyObject_GC_IS_TRACKED(a) after every C-API dict merge. As a
result, once a GC-tracked dict is made immortal and one collection runs, the next C-API
PyDict_Update()/PyDict_Merge() on it aborts a debug build. If the dict is the builtins
module dict, the abort is deterministic at interpreter shutdown (the builtins-restore merge in
finalization).

The two sides each enforce a reasonable invariant; they cannot both hold for an immortal
tracked dict. This is the same fragility class as gh-142975 (frozen objects vs. the FT GC's
refcount validation), one layer over: immortal objects vs. the FT GC's untracking policy.

Reproducer

There is no pure-Python way to create a tracked immortal dict (_Py_SetImmortal() and
PyUnstable_SetImmortal() untrack first), so the reproducer uses a 15-line extension around
the internal _Py_SetImmortalUntracked():

/* _immortal.c */
#include <Python.h>
extern void _Py_SetImmortalUntracked(PyObject *op);

static PyObject *
immortalize_tracked(PyObject *self, PyObject *obj)
{
    _Py_SetImmortalUntracked(obj);   /* immortal, still GC-tracked */
    Py_RETURN_NONE;
}
static PyMethodDef methods[] = {
    {"immortalize_tracked", immortalize_tracked, METH_O, NULL}, {NULL}};
static struct PyModuleDef mod = {PyModuleDef_HEAD_INIT, "_immortal", NULL, -1, methods};
PyMODINIT_FUNC PyInit__immortal(void) { return PyModule_Create(&mod); }

# repro.py  (run with PYTHON_GIL=0; the helper extension predates Py_mod_gil)
import _immortal, gc, builtins

b = vars(builtins)
_immortal.immortalize_tracked(b)     # immortal AND still tracked
print("tracked before collect:", gc.is_tracked(b))   # True
gc.collect()                          # FT GC untracks immortal objects
print("tracked after collect:", gc.is_tracked(b))    # False
# interpreter shutdown -> builtins restore does PyDict_Update() -> abort

Observed output:

tracked before collect: True
tracked after collect: False
python: Objects/dictobject.c:4374: dict_merge_api: Assertion `_PyObject_GC_IS_TRACKED(a)' failed.
Aborted (core dumped)

Any tracked immortal dict shows the state corruption (gc.is_tracked() flips to False after
one collection); the abort fires on the first C-API merge into such a dict — the
Python-level dict.update() method does not abort because dict_update_common() calls
dict_merge() directly, bypassing dict_merge_api() and its assert, and nothing on that path
re-tracks the dict either.

The three code points

1. The GC untracks immortals — Python/gc_free_threading.c, update_refs() (line ~982 at
   ec5f154):

   // Exclude immortal objects from garbage collection
   if (_Py_IsImmortal(op)) {
       op->ob_tid = 0;
       _PyObject_GC_UNTRACK(op);
       gc_clear_unreachable(op);
       return true;
   }
   

2. Dicts must stay tracked — Objects/dictobject.c:4374:

   int res = dict_merge(a, b, override, dupkey);
   assert(_PyObject_GC_IS_TRACKED(a));
   

3. Shutdown makes it deterministic — interpreter finalization restores the builtins
   dict via a C-API merge, so an immortal builtins dict aborts every clean exit.

Why the invariants conflict (and a failed naive fix)

I tried the obvious one-line fix — skip immortal objects in update_refs() without
untracking them. That trades this abort for a different one: the tracked immortal objects then
flow into later GC phases and trip validate_gc_objects (gc_free_threading.c:1119,
"unmerged objects should have ob_tid != 0" class assertions), and under thread churn I also
saw Objects/listobject.c:82: ensure_shared_on_resize asserts. So there are (at least) three
mutually inconsistent expectations:

• the FT GC wants immortal objects out of its heaps entirely (untracked, skipped by every
  phase — untracking is how it guarantees "never scan them again");
• free-threaded dict code wants dict instances always tracked (dict_merge_api asserts it);
• immortalization of a tracked container produces exactly the state neither side can handle.

Resolving this likely needs a decision about which invariant gives way:
(a) the GC skips immortals without untracking and every later phase learns to skip them too;
(b) the GC exempts types that rely on always-tracked (or re-tracks on demand);
(c) the dict assert is relaxed to tracked || immortal — only safe if the tracked bit is not
load-bearing for the shared-dict thread-safety paths, which needs an assessment by the
free-threading owners.

Reachability / impact

Today a tracked immortal dict can only be produced via internal APIs
(_Py_SetImmortalUntracked() is PyAPI_FUNC and extensions do reach for the immortalization
internals). Anything that immortalizes live object graphs — per-interpreter object sharing,
pre-fork copy-on-write immortalization experiments, or a future public immortalization API —
will produce them in bulk, so the conflict is worth resolving before more machinery is built
on top. On release builds the assert is compiled out and the visible symptom is only the
silently flipped tracked bit; whether an untracked dict skips any FT thread-safety path that
keys off the tracked bit is part of the assessment needed here.

Environment

• CPython main @ ec5f154 (3.16.0a0), ./configure --disable-gil --with-pydebug
• Linux x86_64 (WSL2 kernel 6.18, Ubuntu 24.04), gcc 13.3.0
• Not bisected; the update_refs() untrack and the dict_merge_api assert both predate this
  report, so 3.14t is presumably affected as well (untested).

Commencez par le reproducteur et inspectez Python/gc_free_threading.c update_refs() ainsi que Objects/dictobject.c dict_merge_api(), en particulier leurs assertions de tracking. Suivez le merge de restauration de builtins lors du shutdown et les chemins ultérieurs de validation du GC afin de déterminer quelle invariant peut être modifiée sans risque. Le travail est terminé lorsque le scénario tracked-immortal-dict ne corrompt plus le tracking et ne provoque plus l’abandon d’un build de débogage free-threaded, y compris pendant le shutdown de l’interpréteur.