python / python/cpython

Double free / use-after-free in `list.append()` when the list grows under `MemoryError` (`_CALL_LIST_APPEND`)

未关闭
#151,818 6 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

interpreter-core type-crash
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

Crash report

What happened?

When list.append(x) has to grow the list's backing array and that allocation fails (i.e.
under MemoryError), the appended item x is decref'd twice. If x is referenced
elsewhere this is a use-after-free: the interpreter aborts with _Py_NegativeRefcount on a
debug build, or segfaults on a release build, instead of raising a recoverable
MemoryError.

This is reachable under genuine memory pressure (a real RLIMIT_AS reproducer with no test
API is included below), so a program that correctly catches MemoryError can still be left
with a corrupted interpreter.

Reproducer

Deterministic, pure Python, using _testcapi.set_nomemory to fail the grow allocation at a
controlled point:

import _testcapi

class C:
    __slots__ = ("ref",)
    def __init__(self, ref):
        self.ref = ref

def fill():
    items = [C(str(i) + "_unique") for i in range(200)]
    out = []
    for it in items:
        out.append(it.ref)          # CALL_LIST_APPEND; it.ref is also held by the C instance

fill()                              # warm up: specialize out.append(...) to CALL_LIST_APPEND
for start in range(1500):
    _testcapi.set_nomemory(start, start + 1)   # fail one allocation, then resume
    try:
        try:
            fill()
        finally:
            _testcapi.remove_mem_hooks()
    except BaseException:
        pass

On a --with-pydebug build this aborts:

./Include/refcount.h:520: _Py_NegativeRefcount: Assertion failed: object has negative ref count
Fatal Python error: _PyObject_AssertFailed

On a release build it segfaults.

Without any test API (real MemoryError)

The same double-free fires under a genuine allocation failure. With an RLIMIT_AS cap so the
list's grow allocation returns NULL naturally (run on a non-ASan build):

import resource

pool = [object() for _ in range(8_000_000)]      # uniquely-referenced items, built before the cap
warm = []
for i in range(3000):
    warm.append(pool[i])                          # specialize CALL_LIST_APPEND
del warm

cur = int(open("/proc/self/statm").read().split()[0]) * 4096   # current virtual size
resource.setrlimit(resource.RLIMIT_AS, (cur + 24 * 1024 * 1024,) * 2)

out = []
for x in pool:
    out.append(x)                                 # real list_resize failure -> double-free -> SIGSEGV
$ ./python natural.py
Segmentation fault            # faulthandler pins the crash to the `out.append(x)` line

Under the same cap, when the failing allocation is not a list-append grow (e.g. appending
large bytes), Python raises a clean, catchable MemoryError and does not crash — so the
segfault is specific to the buggy append path.

Root cause

In the specialized append bytecode _CALL_LIST_APPEND (Python/bytecodes.c):

op(_CALL_LIST_APPEND, (callable, self, arg -- none, c, s)) {
    ...
    int err = _PyList_AppendTakeRef((PyListObject *)self_o, PyStackRef_AsPyObjectSteal(arg));
    UNLOCK_OBJECT(self_o);
    if (err) {
        ERROR_NO_POP();
    }
    ...
}

arg is stolen via PyStackRef_AsPyObjectSteal and handed to _PyList_AppendTakeRef,
which consumes the reference on every path — including decref'ing the item when the grow
fails (_PyList_AppendTakeRefListResizeif (list_resize(...) < 0) { Py_DECREF(newitem); return -1; }, Objects/listobject.c).

But on that failure the uop takes ERROR_NO_POP(), which jumps to exception handling
without removing arg from the value stack. Since arg's reference was already
consumed, the stale arg stackref is now dangling. The eval loop's exception_unwind then
pops the frame's value stack and PyStackRef_XCLOSEs every slot, closing the stale arg
slot — a second decref of the item.

(Confirmed with ASan on a --with-pymalloc build: the item is freed by PyStackRef_XCLOSE
_PyEval_EvalFrameDefault (the exception_unwind handler); both the item's allocation and
the second, use-after-free decref are visible in the report.)

Suggested fix

_CALL_LIST_APPEND must account for the consumed arg on the error path — once
_PyList_AppendTakeRef has taken the reference, the arg stackref is dead and must not be
left on the value stack for exception_unwind to close. The sibling ops already show the two
correct idioms:

  • the comprehension element-adds (LIST_APPEND, SET_ADD, MAP_ADD) call the same kind of
    steal/*TakeRef helper but use ERROR_IF(...), so the codegen drops the consumed input on
    the error path. Concretely, [x for x in ...] (LIST_APPEND, the same
    _PyList_AppendTakeRef helper) does not crash where lst.append(x)
    (_CALL_LIST_APPEND) does;
  • the consuming call ops (_DO_CALL_FUNCTION_EX, _PY_FRAME_EX) call INPUTS_DEAD(); SYNC_SP(); before ERROR_NO_POP().

I audited the other specialized ops: _CALL_LIST_APPEND is the only one that steals a
stack input and then takes a bare ERROR_NO_POP() without either form of accounting, which
is why it is the lone op affected.

Environment

  • CPython main (3.16.0a0); reproduced on --with-pydebug builds (abort) and release builds
    (segfault), both free-threaded and default GIL.

This report and the reduced reproducers were drafted with the assistance of Claude Code; I
have reviewed and reproduced them.

CPython versions tested on:

CPython main branch, 3.16

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.16.0a0 (heads/main:aec0aed1978, Jun 20 2026, 17:44:00) [Clang 22.1.2 (1ubuntu1)]

Linked PRs
  • gh-151826
  • gh-152114
  • gh-152117

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

从 Python/bytecodes.c 中的 _CALL_LIST_APPEND 开始,跟踪 _PyList_AppendTakeRef 进入 Objects/listobject.c,然后在 debug build 上使用提供的 _testcapi 内存失败脚本重现该问题。比较同类的 LIST_APPEND 和会消耗参数的调用操作,并在 debug build 和 release build 下运行 reproducer,以确认 append 失败会引发 MemoryError,且不会造成 refcount 损坏或崩溃。

由索引模型根据 Issue 内容生成。

评估

技术栈
c, python
领域
backend
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
描述清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。