pybind / pybind/pybind11

[FEAT] Function decorators to specify a conversion policy

Open
#2,924 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

This is a follow up on #2813.
Currently, pybind11 provides the possibility to define a return_value_policy for converting C++ objects to Python when returning from a C++ function. However, when passing objects from C++ to Python via arguments of a Python function object, the conversion policy cannot be easily specified. Particularly, this applies to these important situations:

  • trampoline methods utilizing PYBIND11_OVERRIDE* as in #2915
  • callback function objects as in #1200

In this case, the function's arguments are converted with the hard-coded policy = automatic_reference:
https://github.com/pybind/pybind11/blob/417067eeb8d93e8ed49f3ff580712ae2b9a85259/include/pybind11/pytypes.h#L108-L109
which resolves to reference for pointers and to copy for references. While it was argued several times (e.g. https://github.com/pybind/pybind11/pull/1240#issuecomment-355873711) that pybind11 follows a safe copy strategy by default with the option for users to opt-in for reference passing, this is not easily possible for the above-mentioned use cases.

If one needs passing by reference, one strategy is to explicitly turn reference arguments into pointer arguments as here.
However, I propose to decorate the Python function object:

@pybind11.return_value_policy.reference
def callback(MyObject o):
    o.value = 42

What do you think of this idea, @wjakob, @rwgk, @EricCousineau-TRI, @YannickJadoul?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the conversion call in include/pybind11/pytypes.h at the linked location and the PYBIND11_OVERRIDE* implementation in include/pybind11/pybind11.h. Review follow-up issue #2813 and the callback context from #1200, then determine how the proposed Python decorator should select the argument conversion policy for trampoline and callback functions.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.