pybind / pybind/pybind11

[FEAT] Generate docstrings that are compatible with Sphinx and stubgen

Open
#2,619 1 comment 18 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

docs enhancement signatures
Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

This issue follows on from a discussion at https://github.com/python/mypy/issues/9070

Current Behaviour

Currently, when a function is overloaded in pybind, the docstring generated includes a single, generic type signature at the top (name(*args, **kwargs) -> Any).
Sphinx autodoc will document the signature of the function using this generic type signature (https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_docstring_signature). However it recently added support for reading multiple type signatures from the docstring (https://github.com/sphinx-doc/sphinx/issues/2106). This is apparently an already established convention and it's used in CPython source code (https://github.com/python/cpython/blob/c0f22fb8b3006936757cebb959cee94e285bc503/Objects/rangeobject.c#L156).
Furthermore, stubgen (https://mypy.readthedocs.io/en/stable/stubgen.html) uses this generic function signature, which makes the type stubs that it generates useless (this was the original issue discussed in https://github.com/python/mypy/issues/9070).

New Behaviour

This issue is a feature request to prepend docstrings with all of the overload signatures of the function (unless options.disable_function_signatures() has been called).

To use an example from the pybind11 documentation:

py::class_<Pet>(m, "Pet")
   .def(py::init<const std::string &, int>())
   .def("set", (void (Pet::*)(int)) &Pet::set, "Set the pet's age")
   .def("set", (void (Pet::*)(const std::string &)) &Pet::set, "Set the pet's name");

This produces docstrings that looks like the following:

>>> help(example.Pet)

class Pet(__builtin__.object)
 |  Methods defined here:
 |
 |  __init__(...)
 |      __init__(self: Pet, arg0: str, arg1: int) -> None
 |
 |  set(...)
 |      set(*args, **kwargs) -> Any
 |      Overloaded function.
 |
 |      1. set(self: Pet, arg0: int) -> None
 |
 |      Set the pet's age
 |
 |      2. set(self: Pet, arg0: str) -> None
 |
 |      Set the pet's name

This request is to have the docstrings instead look like the following:

>>> help(example.Pet)

class Pet(__builtin__.object)
 |  Methods defined here:
 |
 |  __init__(...)
 |      __init__(self: Pet, arg0: str, arg1: int) -> None
 |
 |  set(...)
 |      set(self: Pet, arg0: int) -> None
 |      set(self: Pet, arg0: str) -> None
 |      Overloaded function.
 |
 |      1. set(self: Pet, arg0: int) -> None
 |
 |      Set the pet's age
 |
 |      2. set(self: Pet, arg0: str) -> None
 |
 |      Set the pet's name

Furthermore, I think it needs to be possible to have all of the function signatures at the start of the docstring generated but without the section headings. This would allow a user to write the whole docstring as one, but still have the function signatures generated for Sphinx and/or stubgen.

So it should be possible to generate a docstring like this:

>>> help(example.Pet.set)

set(...)
 |      set(self: Pet, arg0: int) -> None
 |      set(self: Pet, arg0: str) -> None
 |      Set attributes on the pet.
 |
 |      When arg0 is an integer, the pet's age is set.
 |      When arg0 is a string, the pet's name is set.

Implementation Details

I think the second change, to generate the signatures without signature headings, would require the addition of a new option. To maintain current behaviour, the options.disable_function_signatures() function could still turn all signatures off. A new option disable_signature_headings() would turn only the section headings off.

So the following code would produce the docstring seen above:

py::options options;
options.disable_signature_headings();

py::class_<Pet>(m, "Pet")
   .def(py::init<const std::string &, int>())
   .def("set", (void (Pet::*)(int)) &Pet::set)
   .def("set", (void (Pet::*)(const std::string &)) &Pet::set, "Set attributes on the pet.\n\nWhen arg0 is an integer, the pet's age is set.\nWhen arg0 is a string, the pet's name is set.");

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the py::options configuration, especially disable_function_signatures(), and the code that assembles overloaded function docstrings. Confirm how signatures and section headings are currently emitted, then verify that the requested output supports Sphinx and stubgen while preserving the existing option to disable all signatures.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
documentation
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.