sphinx-doc / sphinx-doc/sphinx
efficient 2-stage parallel builder?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8k
- Forks
- 2.6k
- PR merge metrics
- No merged PRs in 30d
Description
Due to slowness of Sphinx in buidling of the reference manual of SageMath several years ago we implemented a paralell 2-stage customised Sphinx builder, details may be found in https://trac.sagemath.org/ticket/20080
In a nutshell, it runs Sphinx once to generate intersphinx "object.inv" inventory files and pickle files; these are used in a parallelised 2nd run, where independent builders process subcollections of documents independently, using intersphinx extension to make sure cross-references are maintained; finally, the global index is built.
This gives a many-fold speedup on multi-core CPUs (to know by how much exactly, one would need to create a "standard" Sphinx setup for SageMath, and time it against our builder - but certainly if you have, say, 8 CPUs cores available, the speedup is about 5 to 6-fold, if compared with just using 1 CPU).
As Sphinx evolves, our setup gets harder to maintain (it does not help that several people who originally implemented it drifted away from the project), so we would like to streamline and upgrade our setup, using as much as is available from Sphinx upstream.
Perhaps upstream might be interested in having a similar parallel builder based on this idea, then we could just use it, and do not maintain our builder.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files or tests are named. Start by reviewing the SageMath custom builder described in Trac #20080 and Sphinx's current parallel-building support, then compare a standard reference-manual build with the existing two-stage setup. Done means a concrete, maintainable upstream approach is defined, with its performance and compatibility trade-offs documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100