[BUG]: Repeated gil_scoped_acquire in a persistent C++ thread pool causes a PyThreadState_Clear hotspot on free-threaded CPython 3.14

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

v3.1.0

Problem description

Summary

When py::gil_scoped_acquire is repeatedly constructed inside the hot loop of a persistent C++ thread-pool worker, pybind11 appears to create and destroy a PyThreadState on every iteration.

On a free-threaded CPython 3.14 build, this makes PyThreadState_Clear() and the per-thread mimalloc heap cleanup path a major CPU hotspot.

The same workload on a regular CPython 3.14 build does not show a significant PyThreadState_Clear() cost.

I would like to confirm whether this lifecycle is expected and whether pybind11 should provide or document a way to retain/reuse the thread state for persistent foreign threads without keeping the conventional GIL acquired for the entire worker lifetime.

Environment

• pybind11: 3.1.0
• CPython: 3.14.7
• CPython free-threaded build: 3.14.7t
• Compiler: GCC 12.2.0
• OS: Linux x86_64
• Profiling tool: Linux perf
• Build flags: -O2 -g -fno-omit-frame-pointer
• Worker threads: 4
• Workers pinned to CPU 0-3
• Profiling event: cycles:u
• Sampling frequency: 499 Hz

Runtime configuration was verified as follows:

Regular CPython 3.14:

Py_GIL_DISABLED: 0
sys._is_gil_enabled(): true

Free-threaded CPython 3.14t:

Py_GIL_DISABLED: 1
sys._is_gil_enabled(): false

Workload

The application embeds Python and creates a persistent C++ thread pool.

Each worker executes a long-running task containing a loop. A new py::gil_scoped_acquire is constructed inside each iteration, a Python function is called, and the acquire object is then destroyed.

A simplified version of the worker loop is:

auto worker = [compute, deadline, python_rounds]() {
    std::uint64_t iteration = 0;
    std::uint64_t checksum = 0;

    while (std::chrono::steady_clock::now() < deadline) {
        {
            py::gil_scoped_acquire acquire;

            py::object callable =
                py::reinterpret_borrow<py::object>(compute);

            checksum ^= callable(iteration, python_rounds)
                            .cast<std::uint64_t>();
        }

        ++iteration;
    }

    return checksum;
};

The Python function performs a small CPU-bound integer calculation:

def thread_pool_compute(seed, rounds):
    value = seed & 0xffffffffffffffff
    for i in range(rounds):
        value ^= (value << 13) & 0xffffffffffffffff
        value ^= value >> 7
        value ^= (value << 17) & 0xffffffffffffffff
        value = (
            value + i + 0x9e3779b97f4a7c15
        ) & 0xffffffffffffffff
    return value

The embedding thread releases/detaches its thread state while waiting for the worker tasks:

{
    py::gil_scoped_release release_main_thread;

    ThreadPool pool(thread_count);
    // Submit and wait for worker tasks here.
}

This is required for the regular GIL-enabled CPython build so that worker threads can enter gil_scoped_acquire.

Results

Both variants ran the same workload for 15 seconds with four workers and 32 Python calculation rounds per call.

Metric	CPython 3.14	CPython 3.14t
Python calls	717,248	1,192,960
Throughput	47,810 calls/s	79,447 calls/s
Relative throughput	1.00x	1.66x
Estimated cycles/call	~42.8K	~100.2K
Relative CPU cost/call	1.00x	~2.34x
Lost perf samples	0	0

The free-threaded build improves wall-clock throughput through parallel execution, but the CPU cost per call is substantially higher.

CPU profile

Regular CPython 3.14

gil_scoped_acquire constructor, inclusive:       2.04%
gil_scoped_acquire destructor, inclusive:        1.23%
PyThreadState_Clear, inclusive/self:              0.16%
take_gil, inclusive:                              0.61%

The main self hotspots are Python object allocation and integer operations. For example, _PyObject_Malloc accounts for approximately 22.34%.

Free-threaded CPython 3.14t

gil_scoped_acquire constructor, inclusive:        5.04%
gil_scoped_acquire destructor, inclusive:        44.76%
gil_scoped_acquire::dec_ref, inclusive:           44.73%
PyThreadState_Clear, inclusive:                   43.91%
mi_heap_collect_ex, inclusive:                    38.80%
mi_heap_collect_ex, self:                         37.04%

The dominant call chain is:

pybind11::gil_scoped_acquire::~gil_scoped_acquire
  -> pybind11::gil_scoped_acquire::dec_ref
    -> PyThreadState_Clear
      -> _PyThreadState_ClearMimallocHeaps
        -> _mi_heap_collect_abandon
          -> mi_heap_collect_ex

On AArch64, I also verified that the deeper cleanup path can reach:

mi_heap_collect_ex
  -> _mi_page_free_collect
    -> _mi_page_thread_free_collect
      -> __aarch64_cas8_acq_rel

Suspected cause

For a foreign C++ thread, gil_scoped_acquire creates a new thread state when pybind11 does not find one in its thread-local storage:

if (!tstate) {
    tstate = PyThreadState_New(internals.istate);
    tstate->gilstate_counter = 0;
    internals.tstate = tstate;
}

When the outermost gil_scoped_acquire is destroyed, gilstate_counter reaches zero and pybind11 clears and deletes the thread state:

if (tstate->gilstate_counter == 0) {
    ++tstate->gilstate_counter;
    PyThreadState_Clear(tstate);
    --tstate->gilstate_counter;

    if (active) {
        PyThreadState_DeleteCurrent();
    }

    detail::get_internals().tstate.reset();
    release = false;
}

As a result, using an outermost gil_scoped_acquire inside the loop of a persistent thread-pool worker causes a complete thread-state create/clear/delete cycle on every Python call.

This behavior is especially expensive on CPython 3.14t because clearing the thread state also collects its per-thread mimalloc heaps.

Expected behavior / questions

I understand that keeping a thread state attached indefinitely may have interpreter-finalization and lifetime implications, so I am not sure whether the current behavior is intentional.

Could the maintainers clarify the recommended pattern for persistent foreign worker threads?

In particular:

1. Is repeatedly creating and deleting the PyThreadState for every outermost gil_scoped_acquire expected?
2. Is there a supported way to retain the pybind11-managed thread state for the lifetime of a persistent worker thread while still using scoped Python entry/exit?
3. Should pybind11 provide a separate worker-thread attachment/thread-state guard for this use case?
4. Should this performance characteristic be documented for free-threaded CPython?
5. Could gil_scoped_acquire reuse a previously created thread state until the native thread exits, instead of deleting it whenever the scoped reference count reaches zero?

Moving gil_scoped_acquire outside the worker loop avoids the repeated cleanup on a free-threaded build:

py::gil_scoped_acquire acquire;

while (!done) {
    call_python();
}

However, the same pattern is not generally suitable for a regular CPython build because it causes one worker to retain the GIL for the entire loop. This makes it difficult to use one portable thread-pool pattern for both GIL-enabled and free-threaded builds.

Reproduction steps

# Prepare both Python environments
uv python install 3.14
uv python install 3.14t

uv venv --python 3.14 .venv-314
uv venv --python 3.14t .venv-314t

uv pip install --python .venv-314/bin/python pybind11==3.1.0
uv pip install --python .venv-314t/bin/python pybind11==3.1.0

# Build the embedding executable once against each Python installation,
# using -O2 -g -fno-omit-frame-pointer.

# Profile all worker TIDs
perf record \
    -e cycles:u \
    -F 499 \
    -g \
    --call-graph fp \
    -t <comma-separated-worker-tids> \
    -- sleep 13

perf report

I can provide the complete standalone reproduction project and the two perf.data files if that would be useful.

Reproducible example code

Is this a regression? Put the last known working version here if it is.

Not a regression

Start at the gil_scoped_acquire constructor, destructor, and dec_ref paths shown in the issue, then run the supplied persistent-worker reproduction under CPython 3.14 and 3.14t with perf. Done should establish whether the observed thread-state create/clear/delete cycle is expected and identify the supported retention or documentation change, if any.