pybind / pybind/pybind11

[BUG]: Document MSVC `std::mutex` ABI hazard with older `msvcp140.dll` and `_DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR`

Open Beginner friendly
#6,053 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triage
Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Required prerequisites
What version (or hash if on master) of pybind11 are you using?

3.0.4

Problem description

We recently debugged a real downstream Windows crash where a pybind11-based extension was built with newer MSVC STL headers, but at runtime its std::mutex lock path resolved to an older msvcp140.dll that had been preloaded by another Python wheel.

Microsoft documents _DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR as the compatibility workaround for this newer-toolset / older-redistributable mismatch. It would be useful for pybind11’s Windows build documentation to mention this for projects distributing MSVC-built binary wheels.

The hazard

With newer VS 2022 MSVC STL headers, std::mutex can use a constexpr zero-initialized layout. The constructor may emit no runtime _Mtx_init_in_situ call; later lock operations call _Mtx_lock from msvcp140.dll.

If another wheel in the same Python process has already loaded an older msvcp140.dll under the normal DLL name, the Windows loader may bind the extension’s _Mtx_lock import to that older runtime. The older runtime may expect the pre-constexpr mutex layout, so it can interpret the newer zero-initialized object incorrectly and crash with an access violation.

In the observed case, the first affected lock happened inside pybind11 import-time internals initialization:

PYBIND11_MODULE_PYINIT
  -> pybind11::detail::ensure_internals()
  -> pybind11::detail::get_internals()
  -> internals_pp_manager::create_pp_content_once()
  -> std::lock_guard<std::mutex> lock(pp_set_mutex_)

But the issue class is broader than that one lock: any std::mutex compiled into the extension can be affected if construction and locking are interpreted by incompatible MSVC STL runtime layouts.

Real-world incident
  • Incident: metaopt/optree#278
  • Downstream fix: metaopt/optree#279

The crash dump showed:

msvcp140!mtx_do_lock+0x74
optree._C!PyInit__C+0xa098
optree._C+0x4e29
optree._C!PyInit__C+0x3d

The faulting address was inside pyarrow’s bundled msvcp140.dll 14.28.29334.0. The affected wheel’s import table showed _Mtx_lock / _Mtx_unlock, but not _Mtx_init_in_situ.

After defining _DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR for the extension target, the built .pyd import table included:

_Mtx_lock
_Mtx_init_in_situ
_Mtx_unlock

The original reporter confirmed that the patched wheel imports cleanly in the same environment without the previous import-order workaround.

Suggested documentation text

A note like this could live in the Windows / compiling docs:

MSVC std::mutex compatibility with older msvcp140.dll

Windows extension modules built with newer MSVC STL headers and dynamic CRT linking may be loaded into Python processes where another wheel has already loaded an older msvcp140.dll.

If the extension uses std::mutex, this can produce an MSVC STL mutex ABI mismatch: the extension’s mutex object is constructed using the newer header layout, but _Mtx_lock resolves at runtime to an older msvcp140.dll implementation that expects the older layout. One symptom is an access violation in msvcp140!mtx_do_lock during module import.

Microsoft provides _DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR as a compatibility workaround. Projects affected by this can define it for their extension target:

if(MSVC)
    target_compile_definitions(your_module PRIVATE _DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR)
endif()

For setuptools:

import sys
from setuptools import Extension

ext = Extension(
    'your_module',
    sources=[...],
    define_macros=(
        [('_DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR', None)]
        if sys.platform == 'win32' else []
    ),
)

You can verify that the macro affected the build by inspecting the .pyd import table:

llvm-objdump -p your_module.cp*-win_amd64.pyd | grep _Mtx_

Expected output includes _Mtx_init_in_situ alongside _Mtx_lock / _Mtx_unlock. _Mtx_destroy_in_situ is not expected.

Trade-offs: mutex construction is no longer a constexpr no-op, and constinit std::mutex patterns are disabled inside that target. For typical extension modules this is usually negligible, but projects should apply the macro at the extension-target level rather than globally.

Reproducible example code

Is this a regression? Put the last known working version here if it is.

Not a regression

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Windows or compiling documentation and review the suggested MSVC std::mutex compatibility text in this issue. Document the _DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR workaround for affected extension targets, including the CMake and setuptools examples, and explain verification with llvm-objdump; done when the guidance covers the hazard, trade-offs, and expected import-table symbols.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, python
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.