pybind / pybind/pybind11

[BUG]: Thread heap collapse when compile with option /MTd on Windows msvc

Open
#3,875 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triage
Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Required prerequisites
Problem description

Build two python wrapper package by pybind11 version 2.9.2, and with msvc compiler option /MTd instead of /MDd on Windows.
Then import these two packages will cause thread heap collapse when python exit.
pic

Demo code to reproduce this runtime error.

Some clue:

  1. This runtime error only exists debug build on Windows. Release is OK.
  2. It's ok if switch pybind11 back to 2.5.0 or earlier. The key difference is in pybind11\include\pybind11\detail\class.h::make_default_metaclass(), line 269, which is type->tp_dealloc = pybind11_meta_dealloc. Add a dealloctor to PyTypeObject.
  3. I tried enable macro WITH_THREAD, it failed neither.
Reproducible example code
// C++ code, two simple class
class Test
{
public:
    Test() = default;
    ~Test() = default;
};

class Test2
{
public:
    Test2() = default;
    ~Test2() = default;
};

// bind code, bind two classes to two python pacakges
// bind.cpp
PYBIND11_MODULE(bind, m)
{
    pybind11::class_<Test>(m, "Test")
        .def(py::init<>());
}

// bind2.cpp
PYBIND11_MODULE(bind2, m)
{
    pybind11::class_<Test2>(m, "Test2")
        .def(py::init<>());
}

// use cmake to generate bind.pyd and bind2.pyd with compile option /MTd instead of /MDd
project(pybind_test)
cmake_minimum_required(VERSION 3.20)
find_package(PythonLibs REQUIRED)
include_directories(pybind11/include)
set(CMAKE_BUILD_TYPE Debug)

set(variables
	CMAKE_CXX_FLAGS_DEBUG
	CMAKE_CXX_FLAGS_RELEASE
	CMAKE_CXX_FLAGS_RELWITHDEBINFO
	CMAKE_CXX_FLAGS_MINSIZEREL
   )

foreach(variable ${variables})
	if(${variable} MATCHES "/MD")
		string(REGEX REPLACE "/MD" "/MT" ${variable} "${${variable}}")
	endif()
endforeach()

include_directories(${PYTHON_INCLUDE_DIRS})
add_library(bind SHARED bind.cpp)
set_target_properties(bind PROPERTIES SUFFIX ".pyd")
target_link_libraries(bind ${PYTHON_LIBRARIES})

add_library(bind2 SHARED bind2.cpp)
set_target_properties(bind2 PROPERTIES SUFFIX ".pyd")
target_link_libraries(bind2 ${PYTHON_LIBRARIES})

add_custom_command(OUTPUT output_pyd
   COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_BINARY_DIR}/${CMAKE_BUILD_TYPE}/bind.pyd ${CMAKE_BINARY_DIR}/../pac/bind.pyd
   )
add_custom_command(OUTPUT output_pyd2
   COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_BINARY_DIR}/${CMAKE_BUILD_TYPE}/bind2.pyd ${CMAKE_BINARY_DIR}/../pac/pac2/bind2.pyd
   )
add_custom_target(CopyTask ALL DEPENDS output_pyd bind output_pyd2 bind2)

// when compile finished, reorganize packages like this
pac
├─ bind.pyd
├─ __init__.py
│
└─pac2
   ├─ bind2.pyd
   └─ __init__.py

// content pac/__init__.py is
from .bind import *
from .pac2.bind2 import *

// then run this python script will cause runtime error
import pac as p

t = p.Test()
t2 = p.Test2()

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pybind11/include/pybind11/detail/class.h, especially make_default_metaclass() and the assignment of pybind11_meta_dealloc. Reproduce the two-package example from the linked demo on Windows with MSVC /MTd in a Debug build, then compare behavior with pybind11 2.5.0 and /MDd. Done means the packages can be imported and Python can exit without the thread heap collapse.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.