pybind / pybind/pybind11

Startup deadlock when using multiple interpreters in same process

Open
#1,975 0 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Issue description

When multiple sub-interpreters are created in the same process, the importing of a module created using pybind11 will cause a deadlock, as the GIL for the wrong interpreter is locked. mod_wsgi is the largest example app which will create multiple sub-interpreters, and describe the issue here.

This issue occurs only with pybind11 >= 2.4.0. Older versions do not have this issue (at least as far back as 2.2.3)

Reproducible example code

Code difficult to provide without dependencies, as this is a rather specific integration bug. Most simple way to reproduce is:

  1. Install Apache httpd with mod_wsgi
  2. Create multiple WSGIScriptAlias definitions in apache config file, both pointing to a script which imports a module created with pybind11
  3. At least 1 interpreter will deadlock on import statement.

Attaching the the process reveals that the deadlock occurs here.
Use of PyGILState_Ensure will cause issues in processes with multiple interpreters. If this section is replaces with equivalent code to use of Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS, no issue should occur, as these APIs correctly deal with multiple sub-interpreters.

See reference here for more info on the issue. Specifically, see note of:

Note that the PyGILState_*() functions assume there is only one global interpreter (created automatically by Py_Initialize()). Python still supports the creation of additional interpreters (using Py_NewInterpreter()), but mixing multiple interpreters and the PyGILState_*() API is unsupported.

This implies that the full example code above should be used, instead of PyGILState_*

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at include/pybind11/detail/internals.h around line 240 and review the GIL handling used during module import. Compare the current PyGILState_* usage with the Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS path described in the issue, then validate with multiple sub-interpreters through mod_wsgi. Done means imports no longer deadlock across sub-interpreters.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.