pybind / pybind/pybind11

Updates to `mkdoc.py` for generating `constexpr` trees, speed improvements, etc.?

Open
#1,896 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
18k
Forks
2.3k
Avg merge
5d 17h
Merged PRs (30d)
10

Description

(Writing this now, seeing the merge of #1788 :) )

If anyone is interested, some of the developers / contributors for Drake have developed modifications on top of pybind11s mkdoc.py script to do the following:

Speed up processing. At current, mkdoc.py spawns off separate threads for each header being scanned. Perhaps this works OK for smaller projects, but for ours, I found it best to handle all files within the same invocation (which reduced run-time from ~300s to ~30s at the time).
(FWIW I'm not fully sure if there are other ways to get better performance, or if this was coupled to our other changes.)

Generate a nested constexpr struct which contains documentation for different elements. We've also added modifications to try and disambiguate overloads and have a deterministic ordering of sorts. (At first, by source location and line, and then by a more sophisticated schema.)
One "nicety" (for me at least) is it allows more local scoping of docstring elements, e.g. constexpr auto& cls_doc = my_project_doc.example.ExampleClass, and then referring to things more briefly, like cls_doc.ctor.doc, cls_doc.MyMethod.doc.

Here's our developer documentation:
https://drake.mit.edu/doxygen_cxx/group__python__bindings.html#PydrakeDoc

Our fork of the script:
https://github.com/RobotLocomotion/drake/blob/75fc7b6b9077a9e66a0543352fcaf2653e683d73/third_party/com_github_pybind_pybind11/mkdoc.py

At the time of writing, here's an example of the output:
https://github.com/RobotLocomotion/drake/blob/75fc7b6b9077a9e66a0543352fcaf2653e683d73/tools/workspace/pybind11/test/sample_header_documentation.expected.h

Example Sphinx generation (note that ctors are currently missing from generated docs, which is a known issue):
https://drake.mit.edu/pydrake/index.html#:// (note that the Sphinx doc generation also has some template stuff in it - that's decoupled from this)

If you're interested in seeing some of these changes filter in, please let me know!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing pybind11's mkdoc.py with the Drake fork linked in the issue, then inspect the sample_header_documentation.expected.h output. Determine which speed, constexpr-tree, overload-disambiguation, and ordering changes are suitable for pybind11, and use the existing documentation-generation examples to define the expected result.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
documentation, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.