Deadlock when sub-interpreter throws exception
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 18k
- Forks
- 2.3k
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 10
Description
Issue description
I've been experimenting with a basic usage of sub-interpreters. I face deadlocks as soon as some Python call throws an exception, a simple example is extending the Subinterpreter test in https://github.com/pybind/pybind11/blob/master/tests/test_embed/test_interpreter.cpp#L161, see below.
I'm sure it has to do with the caveat mentioned in https://pybind11.readthedocs.io/en/stable/advanced/embedding.html#sub-interpreter-support, since the deadlock happens in ~error_already_set(), when creating a gil_scoped_acquire object. Is there any easy way around this, or does it require deeper changes in pybind11?
Reproducible example code
TEST_CASE("Subinterpreter") {
// Add tags to the modules in the main interpreter and test the basics.
py::module::import("__main__").attr("main_tag") = "main interpreter";
{
auto m = py::module::import("widget_module");
m.attr("extension_module_tag") = "added to module in main interpreter";
REQUIRE(m.attr("add")(1, 2).cast<int>() == 3);
}
REQUIRE(has_pybind11_internals_builtin());
REQUIRE(has_pybind11_internals_static());
/// Create and switch to a subinterpreter.
auto main_tstate = PyThreadState_Get();
auto sub_tstate = Py_NewInterpreter();
// Subinterpreters get their own copy of builtins. detail::get_internals() still
// works by returning from the static variable, i.e. all interpreters share a single
// global pybind11::internals;
REQUIRE_FALSE(has_pybind11_internals_builtin());
REQUIRE(has_pybind11_internals_static());
// Modules tags should be gone.
REQUIRE_FALSE(py::hasattr(py::module::import("__main__"), "tag"));
{
auto m = py::module::import("widget_module");
REQUIRE_FALSE(py::hasattr(m, "extension_module_tag"));
// Function bindings should still work.
REQUIRE(m.attr("add")(1, 2).cast<int>() == 3);
// This will cause a dead-lock when creating gil_scoped_acquire in ~error_already_set()
try {
py::module::import("widget_modul");
}
catch (const std::exception& e) {
}
}
// Restore main interpreter.
Py_EndInterpreter(sub_tstate);
PyThreadState_Swap(main_tstate);
REQUIRE(py::hasattr(py::module::import("__main__"), "main_tag"));
REQUIRE(py::hasattr(py::module::import("widget_module"), "extension_module_tag"));
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Subinterpreter test in tests/test_embed/test_interpreter.cpp around the referenced example, and review the sub-interpreter caveat in the embedding documentation. Reproduce the exception path involving ~error_already_set() and gil_scoped_acquire. Done means the shown failing case no longer deadlocks while the existing sub-interpreter checks still pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100