Crash in HAMT during garbage collection

Crash report

What happened?

We discovered a crash in the https://github.com/datadog/dd-trace-py library we maintain that actually seems to be rooted in CPython, not our own code.

Error UnixSignal: Process terminated with SEGV_MAPERR (SIGSEGV)
#0  0x00007fc92c3dd27f PyObject_GC_UnTrack
#1  0x00007fc92c4e1448 PyContextVar_Set
#2  0x00007fc92c3f3546 _PyEval_EvalFrameDefault
#3  0x00007fc92c3fbce7 PyObject_Vectorcall
#4  0x00007fc92c3f039b _PyEval_EvalFrameDefault
#5  0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#6  0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#7  0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#8  0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#9  0x00007fc92c44e25f PyIter_Send
#10 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#11 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#12 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#13 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#14 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#15 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#16 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#17 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#18 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#19 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#20 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#21 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#22 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#23 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#24 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#25 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#26 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#27 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#28 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#29 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#30 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#31 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#32 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#33 0x00007fc92c3f2f38 _PyEval_EvalFrameDefault
#34 0x00007fc8f1f854ff __Pyx_PyObject_Call (/project/uvloop/loop.c:191431:15)
#35 0x00007fc8f204f948 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66901:25)
#36 0x00007fc8f2053f7c __pyx_f_6uvloop_4loop_4Loop__on_idle (/project/uvloop/loop.c:17975:25)
#37 0x00007fc8f204f9a6 __pyx_f_6uvloop_4loop_6Handle__run (/project/uvloop/loop.c:66927:24)
#38 0x00007fc8f2051047 __pyx_f_6uvloop_4loop_cb_idle_callback (/project/uvloop/loop.c:87335:19)
#39 0x00007fc8f20687d1 uv__run_idle (/project/build/libuv-x86_64/src/unix/loop-watcher.c:68:1)
#40 0x00007fc8f2065b07 uv_run (/project/build/libuv-x86_64/src/unix/core.c:439:5)
#41 0x00007fc8f1fb83a0 __pyx_f_6uvloop_4loop_4Loop__Loop__run (/project/uvloop/loop.c:18458:23)
#42 0x00007fc8f1ff612d __pyx_f_6uvloop_4loop_4Loop__run (/project/uvloop/loop.c:18876:18)
#43 0x00007fc8f1ff1e85 __pyx_pf_6uvloop_4loop_4Loop_24run_forever (/project/uvloop/loop.c:31528:18)
#44 0x00007fc8f1ff1e85 __pyx_pw_6uvloop_4loop_4Loop_25run_forever (/project/uvloop/loop.c:31331:13)
#45 0x00007fc92c3ec26e PyObject_VectorcallMethod
#46 0x00007fc92c3f039b _PyEval_EvalFrameDefault
#47 0x00007fc92c47474c PyEval_EvalCode
#48 0x00007fc92c4a24a7 _PyRun_SimpleFileObject
#49 0x00007fc92c4a1b38 _PyRun_AnyFileObject
#50 0x00007fc92c49c3be Py_RunMain
#51 0x00007fc92c46461b Py_BytesMain
#52 0x00007fc92c067d65 __libc_start_main
#53 0x0000559068986071 _start

We fixed the crash on our side with https://github.com/DataDog/dd-trace-py/pull/17407 but figured it might still be worth reporting upstream.

The crash happens in PyObject_GC_UnTrack, called from the dealloc path of a HAMT node that PyContextVar_Set is trying to free via Py_DECREF. If that HAMT node's memory is already unmapped, the PyObject_GC_UnTrack write faults.

We discovered that crash in dd-trace-py accidentally because of the following code:

1. Store a fresh {} dict in the context's HAMT under self._storage
2. Put the Token returned by set() back into that same dict

def __enter__(self) -> "BaseWrappingContext":
    token: Token = self._storage.set({})          # 1. create new HAMT entry
    self._storage.get()["__dd_wrapping_context_token__"] = token  # 2. store Token in dict
    return self

Token objects holds tok_ctx, which is a strong reference to the PyContext that was active when set was called -- i.e. ts->context.

I unfortunately don't have a proposed fix yet, but I'm looking into it.

CPython versions tested on:

3.12

EDIT: this is happening on 3.12 but the stack I originally reported (and most crashes I see) are coming from Python 3.10 and 3.11.

Operating systems tested on:

Linux

Linked PRs

• gh-148928

Beginne bei den in der Absturzspur gezeigten Einstiegspunkten PyContextVar_Set und PyObject_GC_UnTrack und überprüfe die verknüpfte Arbeit in gh-148928. Vergleiche die gemeldete HAMT-Deallokationssequenz über CPython 3.10–3.12 hinweg; abgeschlossen ist die Aufgabe, wenn der Absturz für die betroffenen Versionen verstanden und behoben ist.