python / python/cpython

Improve detection of `Py_DECREF` on dictionaries in freelist

Open
#142,280 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

interpreter-core type-feature
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Crash report

What happened?

After updating my extension library from Python 3.11 to 3.14, I had a strage segmentation fault raised by my tests. The analysis of this issue took me about a week. The problem here was the introduction of freelists for Python Dictionaries:

When a Python dictionary is freed (refcnt == 0), it get's appended to a list of "free" objects, which may get reused by subsequent PyDict_New() calls. This freelist is implemented as a linked LIFO list. The link between the elements in this list is done by "reusing" the reference counter of the dictionary and storing the address of the next elements in the list in it. This can be problematic because of two reasons:

First of all, if I accidentally Py_DECREF a dictionary too many times, the reference counter does not become negative and hence, this issue is not detected. This is, because the reference counter will container a pointer to the next element as soon as it gets zero. Hence, the next call of Py_DECREF will read the pointer address as integer and assume it is just very large (or in best case negative, in which case it may fail).
This is bad, as it hardens bug detection in debug builds.

But the real problem araises, if the integer representation of the pointer in this dict is not larget than the minimum reference count for immortals: If the integer is larger than this value, the dict is counted as immortal and nothing happens - happy case. If not, Py_DECREF (or Py_INCREF) will modify the reference count and thus, accidentally modifies the pointer! At this point the pointer is off by one and the freelist is broken.

Now, if two new dictionaries are requested, this can cause a segmentation fault on the second dict, because the process might read one byte behind its memory. In best case, this pops up near to the place, where the dict was decref'd, but in worse case, this situation can linger for a long time in the program and pop up in a completely unrelated place. E.g. even when doing d = {} in the Python code.

The following code tries to demonstrate this problem:

#include <Python.h>

void main() {
    PyObject *d1, *d2, *dict;
    Py_Initialize();

    // populate the freelist with two dicts
    d1 = PyDict_New();
    Py_DECREF(d1);

    d2 = PyDict_New();
    Py_DECREF(d2);

    // Now decref a dict too many times
    // For this, first create a new dict. It will take the first entry from the freelist (d2).
    dict = PyDict_New();
    assert(dict == d2);
    assert(Py_REFCNT(dict) == 1);

    // Now if we decref this dict, the reference counter will be overwritten
    // with the next element in the freelist (d1).
    Py_DECREF(dict);
    assert(Py_REFCNT(dict) == (Py_ssize_t)d1);

    // Now comes the bug: 
    // IFF (int)d1 < _Py_IMMORTAL_MINIMUM_REFCNT, the dict is not considered immortal
    // and a Py_DECREF will accidentally modify the address pointer
    Py_DECREF(dict);
    assert(PyUnstable_IsImmortal(dict) || Py_REFCNT(dict) == (Py_ssize_t)d1 - 1);
}

The output of this depends on the address of d1 and whether it's integer representation is larger than _Py_IMMORTAL_MINIMUM_REFCNT or not.

CPython versions tested on:

3.14

Operating systems tested on:

Linux, Windows

Output from running 'python -VV' on the command line:

Python 3.14.1 (main, Dec 2 2025, 19:40:56) [Clang 21.1.4 ]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the behavior with the C++ example in the issue on CPython 3.14, then trace dictionary freelist handling and Py_DECREF/Py_INCREF behavior in the CPython source. The issue does not name specific files or tests, and completion criteria beyond improving detection of the corruption are not specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.