[BUG]: Python 3.9+ segfault in Debug build for pure virtual error message when GIL released

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.10.1

Problem description

Calling a pure virtual method that is bound via a PYBIND11_OVERRIDE_PURE trampoline implementation, where there is no corresponding Python implementation, calls into pybind11::pybind11_fail in order to throw an exception notifying the user that this pure virtual method has not been overridden.

The pybind11::pybind11_fail function has an assert(!PyErr_Occurred()) line, which obviously only (usually) affects Debug builds.

If the method releases the GIL before the C++ body (py::call_guard<py::gil_scoped_release>{}), then when that assertion is triggered in a Debug build, the CPython tstate is NULL and we get a segfault.

This appears to be caused by a change in CPython https://github.com/python/cpython/pull/17080 / https://bugs.python.org/issue38733 - which indicates this affects Python 3.9+. I.e.

IMHO PyErr_Occurred() must not be called if the GIL is released

[...] It can wait for Python 3.9.

Reproducible example code

Confirmed the following behaves as expected in Python 3.8, but segfaults in Python 3.9

CMakeLists.txt

cmake_minimum_required(VERSION 3.21)
project(segfaulty)
find_package(pybind11 REQUIRED)
pybind11_add_module(segfaulty MODULE)
target_sources(segfaulty PRIVATE segfaulty.cpp)

segfaulty.cpp

#include <pybind11/pybind11.h>

struct PureVirtual {
    virtual ~PureVirtual() = default;
    virtual void method() = 0;
};

struct PyPureVirtual : PureVirtual {
    void method() override {
        PYBIND11_OVERRIDE_PURE(void, PureVirtual, method);
    }
};

PYBIND11_MODULE(segfaulty, mod) {
    namespace py = pybind11;
    py::class_<PureVirtual, PyPureVirtual>{mod, "PureVirtual"}
            .def(py::init<>())
            .def("method", &PureVirtual::method, py::call_guard<py::gil_scoped_release>{});
}

Test, assuming pybind11 and Python are discoverable and Python 3.9 is the discovered Python version

cmake -S . -B build -DCMAKE_BUILD_TYPE=Debug 
cmake --build build
cd build
python3.9 -c "import segfaulty; obj = segfaulty.PureVirtual(); obj.method()"

The yields

Segmentation fault (core dumped)

on Python 3.9, and

Traceback (most recent call last):
  File "<string>", line 1, in <module>
RuntimeError: Tried to call pure virtual function "PureVirtual::method"

on Python 3.8.

Is this a regression? Put the last known working version here if it is.

Not a regression

Start with the CMakeLists.txt and segfaulty.cpp reproducer, then trace PYBIND11_OVERRIDE_PURE into pybind11::pybind11_fail and its PyErr_Occurred assertion. Compare Debug behavior with Python 3.8 and 3.9 while the GIL is released; done means the pure-virtual call reports the RuntimeError without a segmentation fault.