[BUG] Fatal Python error: PyMUTEX_LOCK(gil->mutex) failed

Hi,

I've got a package that uses pybind11 (it's awesome, by the way), and had a few users report the following crash. I've been able to reproduce it myself as well. I've asked on the gitter site and had a good discussion with @quantotto but ultimately we came only down to speculation.

I've reduced the issue to a minimum reproducer, so hopefully this is possible to debug. It seems to be a somewhat hidden issue, doesn't appear in every version of python or compiler.

Issue description

When importing a package built with pybind11, the python libraries fail to load with the following error:

Fatal Python error: PyMUTEX_LOCK(gil->mutex) failed
Python runtime state: unknown

Abort trap: 6

This only seems to appear when using conda's python on Mac OS. I haven't reproduced it elsewhere. Suspect that python is built with clang 10 from conda, while the python package in question is built with clang11, and some incompatibility arises.

Reproducible example code

I am sorry I can not give you a more simple example. I've stripped it down as far as I think I can and still reproduce this.

See the repository here: larcv3-pybind11-example
This uses scikit-build to call cmake and build a package including pybind11-generated python bindings.

Here's a list of instructions to reproduce this:

bash Miniconda3-latest-MacOSX-x86_64.sh #install a fresh conda
source miniconda3/bin/activate # activate it
conda install cmake # install cmake
pip install scikit-build # install scikit-build
git clone https://github.com/coreyjadams/larcv3-pybind11-example.git # clone the example
cd larcv3-pybind11-example/ 
git submodule update --init # clone pybind11 as a submodule
python setup.py build # compile
python setup.py install #install

Then, in a python interpreter you can do:

>>> from larcv import pylarcv
Fatal Python error: PyMUTEX_LOCK(gil->mutex) failed
Python runtime state: unknown

Abort trap: 6

This also appears to be related to this github issue: https://stackoverflow.com/questions/66026520/fatal-python-error-pymutex-lock-pyruntime-ceval-gil-mutex-failed

Start with the linked larcv3-pybind11-example and its macOS Miniconda setup; run the listed conda, scikit-build, CMake, build, install, and import steps to reproduce the PyMUTEX_LOCK crash. Compare the Python and compiler environment with the pybind11 submodule involved; done means identifying a reproducible cause and validating a fix.