Improve detection of `Py_DECREF` on dictionaries in freelist
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 77.2k
- 派生
- 35.9k
- PR 合并指标
- PR 指标待抓取
描述
Crash report
What happened?
After updating my extension library from Python 3.11 to 3.14, I had a strage segmentation fault raised by my tests. The analysis of this issue took me about a week. The problem here was the introduction of freelists for Python Dictionaries:
When a Python dictionary is freed (refcnt == 0), it get's appended to a list of "free" objects, which may get reused by subsequent PyDict_New() calls. This freelist is implemented as a linked LIFO list. The link between the elements in this list is done by "reusing" the reference counter of the dictionary and storing the address of the next elements in the list in it. This can be problematic because of two reasons:
First of all, if I accidentally Py_DECREF a dictionary too many times, the reference counter does not become negative and hence, this issue is not detected. This is, because the reference counter will container a pointer to the next element as soon as it gets zero. Hence, the next call of Py_DECREF will read the pointer address as integer and assume it is just very large (or in best case negative, in which case it may fail).
This is bad, as it hardens bug detection in debug builds.
But the real problem araises, if the integer representation of the pointer in this dict is not larget than the minimum reference count for immortals: If the integer is larger than this value, the dict is counted as immortal and nothing happens - happy case. If not, Py_DECREF (or Py_INCREF) will modify the reference count and thus, accidentally modifies the pointer! At this point the pointer is off by one and the freelist is broken.
Now, if two new dictionaries are requested, this can cause a segmentation fault on the second dict, because the process might read one byte behind its memory. In best case, this pops up near to the place, where the dict was decref'd, but in worse case, this situation can linger for a long time in the program and pop up in a completely unrelated place. E.g. even when doing d = {} in the Python code.
The following code tries to demonstrate this problem:
#include <Python.h>
void main() {
PyObject *d1, *d2, *dict;
Py_Initialize();
// populate the freelist with two dicts
d1 = PyDict_New();
Py_DECREF(d1);
d2 = PyDict_New();
Py_DECREF(d2);
// Now decref a dict too many times
// For this, first create a new dict. It will take the first entry from the freelist (d2).
dict = PyDict_New();
assert(dict == d2);
assert(Py_REFCNT(dict) == 1);
// Now if we decref this dict, the reference counter will be overwritten
// with the next element in the freelist (d1).
Py_DECREF(dict);
assert(Py_REFCNT(dict) == (Py_ssize_t)d1);
// Now comes the bug:
// IFF (int)d1 < _Py_IMMORTAL_MINIMUM_REFCNT, the dict is not considered immortal
// and a Py_DECREF will accidentally modify the address pointer
Py_DECREF(dict);
assert(PyUnstable_IsImmortal(dict) || Py_REFCNT(dict) == (Py_ssize_t)d1 - 1);
}
The output of this depends on the address of d1 and whether it's integer representation is larger than _Py_IMMORTAL_MINIMUM_REFCNT or not.
CPython versions tested on:
3.14
Operating systems tested on:
Linux, Windows
Output from running 'python -VV' on the command line:
Python 3.14.1 (main, Dec 2 2025, 19:40:56) [Clang 21.1.4 ]
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
首先,在 CPython 3.14 上使用 issue 中的 C++ 示例复现该行为,然后跟踪 CPython 源代码中 dictionary freelist 的处理以及 Py_DECREF/Py_INCREF 的行为。该 issue 未指定具体文件或测试,除了改进对损坏的检测之外,也未指定其他完成标准。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- backend
- Issue 类型
- 缺陷
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 25/100