[BUG]: Returning a reference to an element within a vector will be invalidated.
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 18k
- Forks
- 2.3k
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 10
Description
Required prerequisites
- Make sure you've read the documentation. Your issue may be addressed there.
- Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
- Consider asking first in the Gitter chat room or in a Discussion.
What version (or hash if on master) of pybind11 are you using?
2.6.2
Problem description
Lets say I have a Camera class and a vector of Cameras. I bind the vector and I bind the Camera class. Everything is working fine until the user wants to reference a single element within the vector rather than taking a reference to the vector as a whole. Well since the heap memory within a std::vector can change/move depending on whether it shrinks or grows, the reference(or pointer) to that single element will be completely invalidated (specifically we're encountering a "heap-use-after-free" error from AddressSanitizer). Does pybind provide a way to avoid this? I've been searching through the forums, docs and issues tab on github but they seem to only talk about binding stl containers as a whole rather than keeping track of a single element within a container. I don't see anything within the return value policies that address this particular use case either. Any advice would be great. Thanks :)
Reproducible example code
// assume Camera and vector<Camera> is opaquely binded.
// c++ code
class CameraContainer{
public:
// ...
// various other non related methods
Camera& add(const int id) {
cameras.emplace_back(Camera(id));
return cameras.back();
}
private:
std::vector<Camera> cameras
};
py::class_<CameraContainer>(m)
.def(
"add",
&CameraContainer::add,
py::arg("id") = -1,
py::return_value_policy::reference_internal); // this is the problem
// python code
// add one camera at a time and return each camera individually.
cameras = [add_camera(id) for id in range(num_views_per_frame)]
Is this a regression? Put the last known working version here if it is.
Not a regression
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the CameraContainer::add example and its reference_internal return policy, then reproduce the list-comprehension case with AddressSanitizer as described. Read the pybind11 documentation and issue discussion referenced in the report to determine whether element lifetime can be supported; done means the invalid reference behavior is resolved or the supported limitation is clearly established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- api, backend-api-design
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100