return_internal_reference not working for class with virtual function

I wrap the template classes A and B through boost python, and try to cast between them. I implement the toB function and wrap it with return_internal_reference<> to achieve this. However, it doesn't return the B object in python, but returns the "dead" A object, of which member function can't be used anymore.

#include
#include
using namespace boost::python;

struct Bar
{
virtual void hi() { std::cout << "Hi Bar." << std::endl; }
virtual ~Bar() {}
};

template
struct Foo : Bar
{
void set(T v) { val = v; }
T get() { return val; }
private:
T val;
};

using A = Foo;
using B = Foo;

B & toB(A & a) { return *reinterpret_cast(&a);}

BOOST_PYTHON_MODULE(example)
{
class_("A")
.def("set", &A::set)
.def("get", &A::get)
;
class_("B")
.def("set", &B::set)
.def("get", &B::get)
;
def("toB", &toB, return_internal_reference<>());
}
Python execution

It works well if I remove the "virtual" keywords in Bar.