Failure to run module destructor (as weakref callback or as atexit handler) leads to segfault when object is later destroyed too late in CPython's shutdown sequence

Issue description

Consider the following example:

#include <pybind11/pybind11.h>

namespace py = pybind11;

py::object o;

PYBIND11_MODULE(python_example, m) {
  o = py::module::import("matplotlib.mathtext").attr("MathTextParser")("foo");

  py::cpp_function cleanup = [](py::handle weakref) {
    printf("weakref cleanup\n");
    o = py::none{};
    weakref.dec_ref();
  };
  (void) py::weakref(m, cleanup).release();

  py::module::import("atexit").attr("register")(
    py::cpp_function{
      [&]() -> void {
        printf("atexit cleanup\n");
        o = py::none{};
      }
    }
  );
}

Compile e.g. using the python_example template repository, Python 3.6, pybind11 2.2.3; run with Matplotlib 3.0rc1 (for example). Note that the matplotlib.mathtext.MathTextParser class appears to be fairly innocuous -- it directly inherits from object, its __init__ is just

def __init__(self, output): self._output = output.lower()

and it has no __del__. Still, for reasons I don't understand, it is the only class with which I have been able to reproduce the bug described below.

The code above sets a C-level global variable to an instance of MathTextParser, then tries to make sure that the instance is destroyed before CPython shuts down, using both "module destructor" methods currently documented by pybind11 (using atexit is documented in master, not in 2.2.3).

On Windows only, open a Python console and import the python_example module, then close the terminal without first exiting Python. This triggers a "Python has stopped working" error ("A problem caused the program to stop working correctly. Windows will close the program and notify you if a solution is available."). Attaching Visual Studio to the process reveals that this is actually a segfault in subtype_dealloc (typeobject.c), with PyThreadState_GET() returning NULL; the type being deallocated is MathTextParser.

My interpretation of what is happening is that CPython has already torn down some critical utilities at that point (hence PyThreadState_GET returning NULL), but the module destructors have not been called. At some later point, we try to decref the global instance of MathTextParser that was still alive, ultimately leading to the deallocation of the MathTextParser type and the above segfault.

See item #3 of https://github.com/anntzer/mplcairo/issues/6 for the place where this issue was first reported.

Reproducible example code

See above.

Start with the provided pybind11 example and reproduce the shutdown crash on Windows. Read CPython's typeobject.c around subtype_dealloc and compare the weakref and atexit cleanup paths during interpreter shutdown. Done means the example no longer segfaults when the terminal closes, with focused regression coverage if the project identifies a suitable test location.