pybind / pybind/pybind11

[BUG]: Returning a reference to an element within a vector will be invalidated.

Open
#4,450 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triage
Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Required prerequisites
What version (or hash if on master) of pybind11 are you using?

2.6.2

Problem description

Lets say I have a Camera class and a vector of Cameras. I bind the vector and I bind the Camera class. Everything is working fine until the user wants to reference a single element within the vector rather than taking a reference to the vector as a whole. Well since the heap memory within a std::vector can change/move depending on whether it shrinks or grows, the reference(or pointer) to that single element will be completely invalidated (specifically we're encountering a "heap-use-after-free" error from AddressSanitizer). Does pybind provide a way to avoid this? I've been searching through the forums, docs and issues tab on github but they seem to only talk about binding stl containers as a whole rather than keeping track of a single element within a container. I don't see anything within the return value policies that address this particular use case either. Any advice would be great. Thanks :)

Reproducible example code
// assume Camera and vector<Camera> is opaquely binded.

// c++ code
class CameraContainer{
public:
// ...
// various other non related methods


Camera& add(const int id) {
    cameras.emplace_back(Camera(id));
    return cameras.back();
}
private:
std::vector<Camera> cameras
};

py::class_<CameraContainer>(m)
.def(
          "add",
          &CameraContainer::add,
          py::arg("id") = -1,
          py::return_value_policy::reference_internal); // this is the problem

// python code
// add one camera at a time and return each camera individually.
cameras = [add_camera(id) for id in range(num_views_per_frame)]
Is this a regression? Put the last known working version here if it is.

Not a regression

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the CameraContainer::add example and its reference_internal return policy, then reproduce the list-comprehension case with AddressSanitizer as described. Read the pybind11 documentation and issue discussion referenced in the report to determine whether element lifetime can be supported; done means the invalid reference behavior is resolved or the supported limitation is clearly established.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
api, backend-api-design
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.