pybind / pybind/pybind11

[FEAT] Support explicitly redirecting/keeping alive the owning object when exposing a buffer_info for the buffer protocol

Open
#2,693 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

Assume I want to expose a certain class T which needs to support the buffer protocol (because there is client code that calls memoryview on it). Additionally, instances of this class do not actually possess a buffer with the correct info, but instead need to synthesize it on-the-fly when requested (see below for an actual use case).

An example implementation would be e.g.

#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>

namespace py = pybind11;

class T {};

PYBIND11_MODULE(python_example, m) {
    py::class_<T>(m, "T", py::buffer_protocol())
        .def(py::init<>())
        .def_buffer([](T&) { return py::array_t<uint8_t>{{1}}.request(); })
        ;
}

(where the buffer we return is just a size-1 uninitialized array in this case, but you can imagine it would be some more complex computation in reality). Unfortunately, I believe that no reference will be kept on the array when the function return, and the resulting memoryview will end up with a dangling pointer (basically because the buffer_info(Py_buffer* view) constructor silently drops the obj field of the Py_buffer).

Instead, it should be possible for the def_buffer method to return buffer_info that includes information about the exporting object (i.e. Py_buffer.obj). I don't have any opinion as to whether this should be spelt as a separate method (e.g. request_keep_alive) or as an additional parameter to request, or yet some other API. Note that having Py_buffer.obj refer to the array (the "root" exporting object) rather than the T is explicitly supported by the CPython docs (https://docs.python.org/3/c-api/typeobj.html#c.PyBufferProcs.bf_getbuffer "The buffer request is redirected to the root object of the tree. Here, view->obj will be a new reference to the root object.").


The actual use case is to implement the PyBufferRegion API (internal) from Matplotlib, which is used to export sub-regions of a canvas; GUI backends then get a memoryview on the region and forward it to the painting widgets. Unfortunately, when using cairo-based renderers, this entails an additional conversion from cairo's premultiplied alpha format to the unmultiplied alpha format used by certain GUI toolkits such as Tk, as well as possibly byte reordering between ARGB32 and RGBA8888; that is the conversion that I would like to perform in def_buffer.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with pybind11's def_buffer and buffer_info(Py_buffer* view) behavior, then read the linked CPython PyBufferProcs.bf_getbuffer documentation about Py_buffer.obj. Done means a synthesized buffer returned by def_buffer keeps its exporting object alive so a resulting memoryview cannot reference freed data.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, numpy, python
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.