pybind / pybind/pybind11

Deadlock when sub-interpreter throws exception

Open
#1,490 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Issue description

I've been experimenting with a basic usage of sub-interpreters. I face deadlocks as soon as some Python call throws an exception, a simple example is extending the Subinterpreter test in https://github.com/pybind/pybind11/blob/master/tests/test_embed/test_interpreter.cpp#L161, see below.

I'm sure it has to do with the caveat mentioned in https://pybind11.readthedocs.io/en/stable/advanced/embedding.html#sub-interpreter-support, since the deadlock happens in ~error_already_set(), when creating a gil_scoped_acquire object. Is there any easy way around this, or does it require deeper changes in pybind11?

Reproducible example code

TEST_CASE("Subinterpreter") {
    // Add tags to the modules in the main interpreter and test the basics.
    py::module::import("__main__").attr("main_tag") = "main interpreter";
    {
        auto m = py::module::import("widget_module");
        m.attr("extension_module_tag") = "added to module in main interpreter";

        REQUIRE(m.attr("add")(1, 2).cast<int>() == 3);
    }
    REQUIRE(has_pybind11_internals_builtin());
    REQUIRE(has_pybind11_internals_static());

    /// Create and switch to a subinterpreter.
    auto main_tstate = PyThreadState_Get();
    auto sub_tstate = Py_NewInterpreter();

    // Subinterpreters get their own copy of builtins. detail::get_internals() still
    // works by returning from the static variable, i.e. all interpreters share a single
    // global pybind11::internals;
    REQUIRE_FALSE(has_pybind11_internals_builtin());
    REQUIRE(has_pybind11_internals_static());

    // Modules tags should be gone.
    REQUIRE_FALSE(py::hasattr(py::module::import("__main__"), "tag"));
    {
        auto m = py::module::import("widget_module");
        REQUIRE_FALSE(py::hasattr(m, "extension_module_tag"));

        // Function bindings should still work.
        REQUIRE(m.attr("add")(1, 2).cast<int>() == 3);

        // This will cause a dead-lock when creating gil_scoped_acquire in ~error_already_set()
        try {
            py::module::import("widget_modul");
        }
        catch (const std::exception& e) {
        }
    }

    // Restore main interpreter.
    Py_EndInterpreter(sub_tstate);
    PyThreadState_Swap(main_tstate);

    REQUIRE(py::hasattr(py::module::import("__main__"), "main_tag"));
    REQUIRE(py::hasattr(py::module::import("widget_module"), "extension_module_tag"));
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Subinterpreter test in tests/test_embed/test_interpreter.cpp around the referenced example, and review the sub-interpreter caveat in the embedding documentation. Reproduce the exception path involving ~error_already_set() and gil_scoped_acquire. Done means the shown failing case no longer deadlocks while the existing sub-interpreter checks still pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.