Loading python imports from dlopen-loaded C++ library that uses pybind11.

I have the following case :

• an app which does not have any dependency on pybind11, python or whatever.
• a plug-in for this app loaded at run-time with dlopen. This plug-in uses pybind11.

So, basically :

[ app ]  --loads--> [ plugin ] --executes--> [ stuff.py ]

When I want to execute some python code that uses scikit_learn from this plug-in, I get the following error:

terminate called after throwing an instance of 'pybind11::error_already_set'
  what():  ImportError: /usr/lib/python3.6/site-packages/sklearn/__check_build/_check_build.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _Py_NoneStruct
___________________________________________________________________________
Contents of /usr/lib/python3.6/site-packages/sklearn/__check_build:
setup.py                  __init__.py               __pycache__
_check_build.cpython-36m-x86_64-linux-gnu.so
___________________________________________________________________________
It seems that scikit-learn has not been built correctly.

If you have installed scikit-learn from source, please do not forget
to build the package before using it: run `python setup.py install` or
`make` in the source directory.

If you have used an installer, please check that it is suited for your
Python version, your operating system and your platform.

At:
  /usr/lib/python3.6/site-packages/sklearn/__check_build/__init__.py(41): raise_build_error
  /usr/lib/python3.6/site-packages/sklearn/__check_build/__init__.py(46): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(678): exec_module
  <frozen importlib._bootstrap>(665): _load_unlocked
  <frozen importlib._bootstrap>(955): _find_and_load_unlocked
  <frozen importlib._bootstrap>(971): _find_and_load
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap>(1024): _handle_fromlist
  /usr/lib/python3.6/site-packages/sklearn/__init__.py(56): <module>
  <frozen importlib._bootstrap>(219): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(678): exec_module
  <frozen importlib._bootstrap>(665): _load_unlocked
  <frozen importlib._bootstrap>(955): _find_and_load_unlocked
  <frozen importlib._bootstrap>(971): _find_and_load
  <string>(2): <module>

However, this works if I load the python code from my "main" executable instead.

Here's a minimal example :

CMakeLists.txt:

set(CMAKE_BUILD_TYPE Debug)
cmake_minimum_required(VERSION 3.1)
project(blah)
add_subdirectory(pybind11)

add_executable(main main.cpp)
add_library(blah SHARED lib.cpp)

target_link_libraries(main dl)
target_link_libraries(blah pybind11::embed)

main.cpp:

#include <dlfcn.h>
int main()
{
  auto dll = dlopen("libblah.so", RTLD_NOW);
  typedef void (*fun_type)();
  auto fun = reinterpret_cast<fun_type>(dlsym(dll, "run"));
  fun();
}

lib.cpp:

#include <pybind11/embed.h>

extern "C" void run()
{
  pybind11::scoped_interpreter guard{};
  pybind11::exec("import sklearn");
}

running:

$ cmake  --build .
$ LD_LIBRARY_PATH=. ./main

Start by reproducing the failure with CMakeLists.txt, main.cpp, and lib.cpp using the shown cmake build and LD_LIBRARY_PATH command. Inspect how the dlopen-loaded plugin initializes the embedded interpreter and how sklearn is imported; done means the minimal example can load sklearn successfully from the plugin, with the supported behavior documented.