Issue binding protobuf RepeatedField and RepeatedPtrField

OS: Windows 10
Compiler: MSVC 2017 x64
Python: Python 3.6.4 (x64)

I'm trying to bind repeated fields generated by google protocol buffers. The binding code uses a custom type_caster to expose the RepeatedField and RepeatedPtrField types used by the automatically generated classes.

Here's a minimal protobuf definition that uses repeated fields. There's nothing particularly complex or in any way bespoke happening here.

syntax = "proto3";

message Value {
    repeated double data = 1;
}

message ValuesResponse {
    repeated Value values = 1;
}

To expose the repeated fields with pybind11, a specialism of the pybind11::type_caster is defined, and for convenience we create a specialism of the bundled pybind11::list_caster, which already does what we need.

namespace pb = google::protobuf;

namespace pybind11 { namespace detail {
	template <typename Type>
	class type_caster<pb::RepeatedField<Type>> : public list_caster<pb::RepeatedField<Type>, Type> {};

	template <typename Type>
	class type_caster<pb::RepeatedPtrField<Type>> : public list_caster<pb::RepeatedPtrField<Type>, Type> {};
}} // namespace pybind11::detail

Now that pybind11 knows how to bind RepeatedField and RepeatedPtrField as lists we can bind these as read-only properties of the respective classes in the module definition. Unfortunately some awkward casts are required here to help the compiler figure out the correct function overload to use.

namespace py = pybind11;

PYBIND11_MODULE(mymodule, m) {
    /* ... SNIP ... */
	py::class_<Value>(m, "Value")
		.def_property_readonly("data", (const pb::RepeatedField<double>&(Value::*)() const) &Value::data)
	;
	py::class_<ValuesResponse>(m, "ValuesResponse")
		.def_property_readonly("values", (const pb::RepeatedPtrField<Value>&(ValuesResponse::*)() const) &ValuesResponse::values)
	;
    /* ... SNIP ... */
}

Now some simple Python code to exercise the module:

import mymodule

# first, a native method (not shown) is invoked that returns an initialised ValuesResponse
response = get_values()

# a debugger watch shows that response.values is fully populated with valid data at this point in the flow of execution

for v in response.values:
    # now the fields of v are empty, and the entire list of response.values is corrupt
    for d in v.data:
        pass # when stepping over using the debugger, there is nothing in v.data any longer

Everything works as expected at first, the call to get_values() returns an instance of an object that appears to have a list of instances of Value and all fields appear to be initialised, but it seems like the bound objects get corrupted / all fields reset to 0 on subsequent reads of the bound values property (note: even len(response.values) will reset the list).

If I had to make a stab at a logical explanation, I would guess this is an issue of object lifetime, and perhaps Python reference couting is to blame. That's as far as I've got, but if anyone can shed further light on what's happening, I am very much open to ideas and inspiration.

Please let me know if a minimal working / broken example is required to reproduce the issue more easily.

Start by reproducing the Windows/MSVC failure with the protobuf Value and ValuesResponse definitions, the custom type_caster specializations, and the read-only properties shown in the issue. Investigate why reading response.values invalidates or resets the bound objects; done means identifying the lifetime or binding defect and adding a regression test for repeated scalar and message fields.