tracemalloc: Heap-use-after-free in _Py_IsImmortal when destroying subinterpreters while tracing

Crash report

What happened?

from concurrent import interpreters
import tracemalloc

# Start tracking memory allocations.
# Tracemalloc captures filenames of code objects loaded during execution.
tracemalloc.start()

# Create a subinterpreter.
# During initialization (specifically _Py_Get_Getpath_CodeObject), 
# the subinterpreter allocates and interns strings (filenames).
# Tracemalloc records pointers to these strings.
interp_id = interpreters.create()

# Ensure the interpreter is fully initialized/used.
interpreters.run_string(interp_id, "pass")

# Destroy the subinterpreter.
# This clears the subinterpreter's interned dictionary, freeing the strings 
# that tracemalloc is still tracking.
interpreters.destroy(interp_id)

# When the script exits, _Py_Finalize calls tracemalloc_deinit.
# Tracemalloc iterates its traces and tries to access the filename strings 

<sys>:0: RuntimeWarning: remaining subinterpreters; close them with Interpreter.close()
=================================================================
==2128302==ERROR: AddressSanitizer: heap-use-after-free on address 0x6f04cd8396e0 at pc 0x569c61046bb6 bp 0x7ffd6872cba0 sp 0x7ffd6872cb98
READ of size 4 at 0x6f04cd8396e0 thread T0
    #0 0x569c61046bb5 in _Py_IsImmortal /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Include/refcount.h:129:12
    #1 0x569c61046bb5 in Py_DECREF /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Include/refcount.h:414:9
    #2 0x569c61046bb5 in tracemalloc_clear_filename /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/tracemalloc.c:706:5
    #3 0x569c60eba2e0 in _Py_hashtable_destroy_entry /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/hashtable.c:382:9
    #4 0x569c60eba2e0 in _Py_hashtable_clear /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/hashtable.c:398:13
    #5 0x569c61047edf in tracemalloc_clear_traces_unlocked /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/tracemalloc.c:721:5
    #6 0x569c61047edf in _PyTraceMalloc_Stop /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/tracemalloc.c:877:5
    #7 0x569c61048f57 in tracemalloc_deinit /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/tracemalloc.c:783:5
    #8 0x569c61048f57 in _PyTraceMalloc_Fini /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/tracemalloc.c:1261:5
    #9 0x569c60fe7fd6 in _Py_Finalize /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/pylifecycle.c:2301:5
    #10 0x569c6106a07a in Py_RunMain /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Modules/main.c:774:9
    #11 0x569c6106b943 in pymain_main /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Modules/main.c:802:12
    #12 0x569c6106baa2 in Py_BytesMain /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Modules/main.c:826:12
    #13 0x72a4ce73bd8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #14 0x72a4ce73be3f in __libc_start_main csu/../csu/libc-start.c:392:3
    #15 0x569c60884e94 in _start (/home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/python+0x1fce94) (BuildId: 0e20bf7695762228d93d1548a3b79cafed8ba475)

0x6f04cd8396e0 is located 0 bytes inside of 52-byte region [0x6f04cd8396e0,0x6f04cd839714)
freed by thread T0 here:
    #0 0x569c609296e6 in free (/home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/python+0x2a16e6) (BuildId: 0e20bf7695762228d93d1548a3b79cafed8ba475)
    #1 0x569c6104789a in tracemalloc_free /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/tracemalloc.c:643:5
    #2 0x569c60c867e1 in unicode_dealloc /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/unicodeobject.c:1704:5
    #3 0x569c60b77d8d in _Py_Dealloc /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/object.c:3207:5
    #4 0x569c60b3e57a in Py_DECREF /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Include/refcount.h:420:9
    #5 0x569c60b3e57a in Py_XDECREF /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Include/refcount.h:513:9
    #6 0x569c60b3e57a in dictkeys_decref /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/dictobject.c:461:17
    #7 0x569c60c8c6ce in clear_interned_dict /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/unicodeobject.c:317:13
    #8 0x569c60c8c6ce in _PyUnicode_ClearInterned /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/unicodeobject.c:14436:5
    #9 0x569c60febf65 in finalize_interp_types /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/pylifecycle.c:1907:5
    #10 0x569c60febf65 in finalize_interp_clear /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/pylifecycle.c:1951:5
    #11 0x569c60fe8fc6 in Py_EndInterpreter /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/pylifecycle.c:2611:5
    #12 0x569c60fe9701 in finalize_subinterpreters /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/pylifecycle.c:2675:9
    #13 0x569c60fe9701 in make_pre_finalization_calls /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/pylifecycle.c:2133:13
    #14 0x569c60fe7e62 in _Py_Finalize /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/pylifecycle.c:2181:5
    #15 0x569c6106a07a in Py_RunMain /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Modules/main.c:774:9
    #16 0x569c6106b943 in pymain_main /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Modules/main.c:802:12
    #17 0x569c6106baa2 in Py_BytesMain /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Modules/main.c:826:12
    #18 0x72a4ce73bd8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

previously allocated by thread T0 here:
    #0 0x569c60929984 in malloc (/home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/python+0x2a1984) (BuildId: 0e20bf7695762228d93d1548a3b79cafed8ba475)
    #1 0x569c6104afa6 in tracemalloc_alloc /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/tracemalloc.c:518:15
    #2 0x569c60c39be3 in PyUnicode_New /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/unicodeobject.c:1320:24
    #3 0x569c60c40520 in _PyUnicode_FromUCS1 /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/unicodeobject.c:2164:11
    #4 0x569c60c40520 in PyUnicode_FromKindAndData /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/unicodeobject.c:2280:16
    #5 0x569c60fa147c in r_object /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/marshal.c:1309:17
    #6 0x569c60fa0f8a in r_object /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/marshal.c:1570:24
    #7 0x569c60fa0e06 in r_object /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/marshal.c:1382:18
    #8 0x569c60fa0f2f in r_object /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/marshal.c:1558:22
    #9 0x569c60f9a792 in read_object /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/marshal.c:1718:9
    #10 0x569c60f9a584 in PyMarshal_ReadObjectFromString /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/marshal.c:1837:14
    #11 0x569c60ec0d18 in unmarshal_frozen_code /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/import.c:3130:20
    #12 0x569c60ec87af in _imp_get_frozen_object_impl /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/import.c:4614:25
    #13 0x569c60ec87af in _imp_get_frozen_object /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/clinic/import.c.h:285:20
    #14 0x569c60de8f25 in _PyEval_EvalFrameDefault /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/generated_cases.c.h:2582:38
    #15 0x569c60dd5514 in _PyEval_EvalFrame /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Include/internal/pycore_ceval.h:118:16
    #16 0x569c60dd5514 in _PyEval_Vector /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/ceval.c:2094:12
    #17 0x569c60a78bfb in _PyObject_VectorcallTstate /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Include/internal/pycore_call.h:136:11
    #18 0x569c60a78bfb in object_vacall /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/call.c:823:14
    #19 0x569c60a78706 in PyObject_CallMethodObjArgs /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/call.c:890:24
    #20 0x569c60ec30ef in import_find_and_load /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/import.c:3807:11
    #21 0x569c60ec30ef in PyImport_ImportModuleLevelObject /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/import.c:3888:15
    #22 0x569c60e126de in _PyEval_ImportName /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/ceval.c:2912:16
    #23 0x569c60ddb155 in _PyEval_EvalFrameDefault /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/generated_cases.c.h:6373:31
    #24 0x569c60dd4afd in _PyEval_EvalFrame /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Include/internal/pycore_ceval.h:118:16
    #25 0x569c60dd4afd in _PyEval_Vector /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/ceval.c:2094:12
    #26 0x569c60dd4afd in PyEval_EvalCode /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/ceval.c:673:21
    #27 0x569c60dcaba4 in builtin_exec_impl /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/bltinmodule.c:1193:17
    #28 0x569c60dcaba4 in builtin_exec /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/clinic/bltinmodule.c.h:579:20
    #29 0x569c60dd7482 in _Py_BuiltinCallFastWithKeywords_StackRefSteal /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/ceval.c:846:11
    #30 0x569c60dde10a in _PyEval_EvalFrameDefault /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/generated_cases.c.h:2391:35
    #31 0x569c60dd5514 in _PyEval_EvalFrame /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Include/internal/pycore_ceval.h:118:16
    #32 0x569c60dd5514 in _PyEval_Vector /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/ceval.c:2094:12
    #33 0x569c60a78bfb in _PyObject_VectorcallTstate /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Include/internal/pycore_call.h:136:11
    #34 0x569c60a78bfb in object_vacall /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/call.c:823:14
    #35 0x569c60a78706 in PyObject_CallMethodObjArgs /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/call.c:890:24
    #36 0x569c60ec30ef in import_find_and_load /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/import.c:3807:11
    #37 0x569c60ec30ef in PyImport_ImportModuleLevelObject /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/import.c:3888:15
    #38 0x569c60dc806e in builtin___import___impl /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/bltinmodule.c:285:12
    #39 0x569c60dc806e in builtin___import__ /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Python/clinic/bltinmodule.c.h:110:20
    #40 0x569c60a76054 in _PyObject_VectorcallTstate /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Include/internal/pycore_call.h:136:11
    #41 0x569c60a76054 in _PyObject_CallFunctionVa /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/call.c:552:18
    #42 0x569c60a75d74 in PyObject_CallFunction /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Objects/call.c:574:14

SUMMARY: AddressSanitizer: heap-use-after-free /home/fuzz/WorkSpace/FusionFuzzLoop/projects/cpython/cpython/build/../Include/refcount.h:129:12 in _Py_IsImmortal
Shadow bytes around the buggy address:
  0x6f04cd839400: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fa
  0x6f04cd839480: fa fa fa fa fd fd fd fd fd fd fd fa fa fa fa fa
  0x6f04cd839500: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
  0x6f04cd839580: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fa
  0x6f04cd839600: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
=>0x6f04cd839680: fd fd fd fd fd fd fd fa fa fa fa fa[fd]fd fd fd
  0x6f04cd839700: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fa
  0x6f04cd839780: fa fa fa fa fd fd fd fd fd fd fd fd fa fa fa fa
  0x6f04cd839800: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
  0x6f04cd839880: fd fd fd fa fa fa fa fa fd fd fd fd fd fd fd fa
  0x6f04cd839900: fa fa fa fa fd fd fd fd fd fd fd fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==2128302==ABORTING

related: #134604

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Start with Python/tracemalloc.c, especially tracemalloc_clear_filename and tracemalloc_deinit, then inspect the cleanup order in Objects/unicodeobject.c and Python/pylifecycle.c. Run the supplied subinterpreter and tracemalloc reproducer under AddressSanitizer. Done means interpreter shutdown no longer accesses freed filename strings and the reproducer completes without a heap-use-after-free.