[BUG]: tp_traverse occasionally executed before C++ ctor
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 18k
- Forks
- 2.3k
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 10
Description
Required prerequisites
- Make sure you've read the documentation. Your issue may be addressed there.
- Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
- Consider asking first in the Gitter chat room or in a Discussion.
What version (or hash if on master) of pybind11 are you using?
2.11.1
Problem description
Enabling Py_TPFLAGS_HAVE_GC in custom_type_setup will occasionally call type->tp_traverse before the C++ ctor. Depending on what the C++ object contains, this can be anything from harmless (if the allocation is zero filled and traversing it just visits some nulls), to segfault (if all-zeroes is not a legal state for the object, for example trying to iterate a libstdc++ std::unordered_map), or worse (if the allocation is not zero filled, the worst case is a security hole).
I don't know if this is a bug in pybind11, Python itself (calling _PyObject_GC_TRACK in PyType_GenericAlloc in Objects/typeobject.c seems kinda suspicious to me), or in my code, but if it's the latter, the same bug also exists at https://pybind11.readthedocs.io/en/stable/advanced/classes.html#custom-type-setup .
Reproducible example code
#include <pybind11/pybind11.h>
#include <unordered_set>
namespace py = pybind11;
class funny_class;
std::unordered_set<funny_class*> items_that_exist;
class funny_class {
public:
funny_class() { items_that_exist.insert(this); }
~funny_class() { items_that_exist.erase(this); }
};
PYBIND11_MODULE(example, m) {
py::class_<funny_class> elem_container(m, "fun",
py::custom_type_setup([](PyHeapTypeObject* heap_type) {
PyTypeObject* type = &heap_type->ht_type;
type->tp_flags |= Py_TPFLAGS_HAVE_GC;
type->tp_traverse = [](PyObject* self_base, visitproc visit, void* arg) {
auto& self = py::cast<funny_class&>(py::handle(self_base));
if (!items_that_exist.count(&self))
puts("ERROR: Item was traversed without being constructed.");
return 0;
};
type->tp_clear = [](PyObject* self_base) {
auto& self = py::cast<funny_class&>(py::handle(self_base));
if (!items_that_exist.count(&self))
puts("ERROR: Item was erased without being constructed.");
return 0;
};
}));
elem_container.def(py::init<>());
}
import example
a = [(example.fun(), example.fun())[1] for _ in range(10000)]
Is this a regression? Put the last known working version here if it is.
Not a regression
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the custom_type_setup example in the issue and reproduce it using the provided C++ module and Python loop. Compare pybind11's type setup with Python's PyType_GenericAlloc and Objects/typeobject.c, then establish whether traversal before construction is caused by pybind11 or Python. Done means the reproduction no longer traverses an unconstructed object, with a regression test or documented upstream limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100