pybind / pybind/pybind11

hiding symbols breaking wrappers on macOS

Open
#1,503 0 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Issue description

Building our pybind11 wrappers with hidden symbols, as is now recommended, is breaking our code on macOS. I think the problem is basically typeid of shared pointers and have a simple example that shows it (I posted a more complex example as a question as issue 1497)

Reproducible example code

a.h (the header file):

#include <memory>
#include <typeinfo>

class __attribute__ ((visibility("default"))) ClassA {
public:
    ClassA(void) {};
    std::type_info const& type_a() const;
    std::type_info const& type_ptr_a() const;
    std::type_info const& inline_type_a() const { return typeid(ClassA); }; 
    std::type_info const& inline_type_ptr_a() const { return typeid(std::shared_ptr<ClassA>); }; 
};

a.cc (source code for a shared library):

#include "a.h"
std::type_info const& ClassA::type_a() const { return typeid(ClassA); };
std::type_info const& ClassA::type_ptr_a() const { return typeid(std::shared_ptr<ClassA>); };

apy.cc (pybind11 wrapper):

#include <typeinfo>
#include "pybind11/pybind11.h"
#include "a.h"

namespace py = pybind11;

PYBIND11_MODULE(apy, mod) {
    py::class_<std::type_info>(mod, "TypeInfo")
            .def("__eq__",
                 [](std::type_info const& self, std::type_info const& other) { return self == other; })
            .def("__ne__",
                 [](std::type_info const& self, std::type_info const& other) { return self != other; })
            .def("name", &std::type_info::name)
            .def("hash_code", &std::type_info::hash_code);

    py::class_<ClassA, std::shared_ptr<ClassA>>(mod, "ClassA")
        .def(py::init<>())
        .def("type_a", &ClassA::type_a, py::return_value_policy::reference)
        .def("type_ptr_a", &ClassA::type_ptr_a, py::return_value_policy::reference)
        .def("inline_type_a", &ClassA::inline_type_a, py::return_value_policy::reference)
        .def("inline_type_ptr_a", &ClassA::inline_type_ptr_a, py::return_value_policy::reference);
}

CmakeLists.txt (this assumes all the files above are in a subdirectory src and that pybind11 is in a subdirectory pybind11, just like the standard pybind11 cmake example:

cmake_minimum_required(VERSION 2.8.12)
add_subdirectory(pybind11)
set(CMAKE_CXX_STANDARD 14)
if (APPLE)
    cmake_policy(SET CMP0042 NEW)
endif (APPLE)
include_directories(pybind11/include)
add_library(a SHARED src/a.cc)
pybind11_add_module(apy src/apy.cc)
target_link_libraries(apy PRIVATE a)

test.py (code that shows the error)

from apy import ClassA
a = ClassA()
assert a.type_a().name() == a.inline_type_a().name()
assert a.type_a() == a.inline_type_a()
assert a.type_ptr_a().name() == a.inline_type_ptr_a().name()
assert a.type_ptr_a() == a.inline_type_ptr_a()

On macOS 10.13.6 using XCode 9.4.1 the last assert fails. The same code works fine on linux with gcc. The same code also works on macOS if I build with symbols visible, e.g. by using make VERBOSE=1, copying the commands and removing -fvisibility=hidden and executing that.

We also have trouble with our persistence framework and we think it's the same problem in disguise. The final step in un-persisting objects is to dynamically cast them to shared_ptr of the original type, and this silently fails. I have not yet tried to make a trivial example of that. It is a much harder problem to work around than the code shown above.

Is there some way to make typeid(std::shared_ptr<T>) reliably give the same answer in a shared library and in pybind11 wrappers while building with -fvisibility=hidden?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Build the reproducer from a.h, a.cc, apy.cc, CMakeLists.txt, and test.py on macOS with hidden symbols, then compare the failing typeid assertions with a visible-symbol build. Done would require establishing a supported way for the shared_ptr type information to remain consistent across the shared library and pybind11 wrapper, with the reproducer passing under hidden visibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, python
Domain
backend-api-design
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.