sphinx-doc / sphinx-doc/sphinx

"env-merge-info" for the "write" phase

Open
#9,480 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

api type:enhancement
Dominant language
Python
Stars
8k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

Is your feature request related to a problem? Please describe.

There are currently no built-in facilities for the parallel-write-safety of extensions which need to collect data during writing and make use of that information later (during generation of output file or build-finished.

The specific issue here is related to the sphinx-sitemap extension, which basically does all its work during the writing phase: it collects data during html-page-context and writes out its production during built-finished. As a result it currently does not work at all in parallel_write mode, and it's not entirely clear how to cleanly make it parallel_write_safe. Which is a shame as that means sphinx-sitemap incurs a large build time wallclock regression when enabled (and also a bright red warning from Sphinx).

Describe the solution you'd like

An event similar to env-merge-info but taking place after the parallel writing phase. It's also not entirely clear whether "15. Generate output files" is parallelised or not, and if so how. If the output file generation is parallelised, there may need to be two different events (one to gather information after parallel post-transforms and doctree-resolution, and an other after output generation).

Describe alternatives you've considered

That Sphinx provides communication pipelines between the workers (and the controller) instead, creating those pipelines by hand is pretty error-prone and heavy especially as the specific parallelisation model is undefined (for good reasons), and will (hopefully) change in the future to better accomodate Windows and macOS.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the env-merge-info, html-page-context, and build-finished events through the parallel writing phase and the output-file generation stage. Clarify which stages are parallelized and define the event behavior needed for extensions such as sphinx-sitemap to collect data safely. Done means the event design and parallel-write safety semantics are agreed and documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.