pybind / pybind/pybind11

[BUG]: Determine if the interpreter is running

Open
#3,690 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triage
Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Required prerequisites
Problem description

Hi, I have an application that uses gtest+pybind11 to test some API calls in a library, but that library also needs to use pybind11 in some calls outside of the test framework. I used to have the scoped_interpreter created in the gtest main right before the tests start, and it would go out of scope when they finished. The problem is when using the library when not in test mode I need certain functions to start the interpreter. If I have them do it right now, I get an error that an interpreter is already running from the test suite when I run it. I also tried to make the scoped_interpreter a static class member for my pybind11 wrapper, and that almost works, except it segfaults when the application exits:

#0  0x00007fffc2dbd675 in ?? () from /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff7ccf813 in PyBytes_FromString () from /usr/local/bin/miniconda/lib/libpython3.8.so.1.0
#2  0x00007fff0b2075f4 in __pyx_pw_13cupy_backends_4cuda_3api_6driver_15CUDADriverError_1__init__(_object*, _object*, _object*) () from /usr/local/bin/miniconda/lib/python3.8/site-packages/cupy_backends/cuda/api/driver.cpython-38-x86_64-linux-gnu.so
#3  0x00007ffff7cd44df in _PyObject_MakeTpCall () from /usr/local/bin/miniconda/lib/libpython3.8.so.1.0
#4  0x00007ffff7cd5a2c in method_vectorcall () from /usr/local/bin/miniconda/lib/libpython3.8.so.1.0
#5  0x00007ffff7cd648f in PyVectorcall_Call () from /usr/local/bin/miniconda/lib/libpython3.8.so.1.0
#6  0x00007ffff7cfc9ac in slot_tp_init.lto_priv () from /usr/local/bin/miniconda/lib/libpython3.8.so.1.0
#7  0x00007ffff7cf7917 in type_call.lto_priv () from /usr/local/bin/miniconda/lib/libpython3.8.so.1.0
#8  0x00007fff0b200f9f in __Pyx__PyObject_CallOneArg(_object*, _object*) () from /usr/local/bin/miniconda/lib/python3.8/site-packages/cupy_backends/cuda/api/driver.cpython-38-x86_64-linux-gnu.so
#9  0x00007fff0b2014d6 in __pyx_f_13cupy_backends_4cuda_3api_6driver_check_status(int, int) [clone .isra.21] [clone .part.22] () from /usr/local/bin/miniconda/lib/python3.8/site-packages/cupy_backends/cuda/api/driver.cpython-38-x86_64-linux-gnu.so
#10 0x00007fff0b20348f in __pyx_f_13cupy_backends_4cuda_3api_6driver_moduleUnload(long, int) () from /usr/local/bin/miniconda/lib/python3.8/site-packages/cupy_backends/cuda/api/driver.cpython-38-x86_64-linux-gnu.so
#11 0x00007fff0b38bc90 in __pyx_tp_dealloc_4cupy_4cuda_8function_Module(_object*) () from /usr/local/bin/miniconda/lib/python3.8/site-packages/cupy/cuda/function.cpython-38-x86_64-linux-gnu.so
#12 0x00007fff0b38a82e in __pyx_tp_dealloc_4cupy_4cuda_8function_Function(_object*) () from /usr/local/bin/miniconda/lib/python3.8/site-packages/cupy/cuda/function.cpython-38-x86_64-linux-gnu.so
#13 0x00007ffff7de77b6 in free_keys_object () from /usr/local/bin/miniconda/lib/libpython3.8.so.1.0
#14 0x00007ffff7de7938 in dict_clear.lto_priv () from /usr/local/bin/miniconda/lib/libpython3.8.so.1.0
#15 0x00007fff19a9bc96 in __pyx_pw_4cupy_5_util_3clear_memo(_object*, _object*) () from /usr/local/bin/miniconda/lib/python3.8/site-packages/cupy/_util.cpython-38-x86_64-linux-gnu.so
#16 0x00007ffff7e56d04 in cfunction_vectorcall_NOARGS () from /usr/local/bin/miniconda/lib/libpython3.8.so.1.0
#17 0x00007ffff7cd648f in PyVectorcall_Call () from /usr/local/bin/miniconda/lib/libpython3.8.so.1.0
#18 0x00007ffff7e3ce84 in atexit_callfuncs () from /usr/local/bin/miniconda/lib/libpython3.8.so.1.0
#19 0x00007ffff7e04fb3 in Py_FinalizeEx.part.28 () from /usr/local/bin/miniconda/lib/libpython3.8.so.1.0
#20 0x0000555555de0da6 in pybind11::finalize_interpreter () at /repro/MatX/build/_deps/pybind11-src/include/pybind11/embed.h:160
#21 0x0000555555de0eb2 in pybind11::scoped_interpreter::~scoped_interpreter (this=0x55555d8d2311 <matx::detail::MatXPybind::Init()::gil>, __in_chrg=<optimized out>) at /repro/MatX/build/_deps/pybind11-src/include/pybind11/embed.h:194
#22 0x00007fffc2c7ba27 in ?? () from /usr/lib/x86_64-linux-gnu/libc.so.6
#23 0x00007fffc2c7bbe0 in exit () from /usr/lib/x86_64-linux-gnu/libc.so.6
#24 0x00007fffc2c590ba in __libc_start_main () from /usr/lib/x86_64-linux-gnu/libc.so.6

Is there a way to determine if the interpreter is already running so I don't start it again? I also tried catching the exception thrown, but that still prints the error that the interpreter is already running as well. Thanks!

Reproducible example code

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with include/pybind11/embed.h around finalize_interpreter and scoped_interpreter, the entry points shown in the stack trace. Review the gtest-based interpreter lifecycle described in the report and determine the supported way to detect or share an already-running interpreter without causing shutdown-time failures; done should include a documented resolution and a reproducible test case.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.