sphinx-doc / sphinx-doc/sphinx

Allow setting maxbatch to adjust chunk size for parallel read/write

Open
#10,967 3 comments 8 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

internals:parallel
Dominant language
Python
Stars
8k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

Is your feature request related to a problem? Please describe.
I'm currently working on a project with a large documentation (~22k files).
We noticed that reading in parallel is particularly slow:

sphinx-build init        0:00:47.712593
sphinx-build read        0:09:39.427063
sphinx-build write       0:03:55.736320
sphinx-build copy, dump  0:04:10.119199

I managed to track this issue down to the calculation of the chunk size for parallel processing:
https://github.com/sphinx-doc/sphinx/blob/cc314f13e8a98393ab018d83d8957a724a6f338a/sphinx/util/parallel.py#L137-L150
When setting maxbatch from 10 to 1000 the read performance improves significantly:

sphinx-build init        0:00:45.914173
sphinx-build read        0:00:50.814347
sphinx-build write       0:03:36.089072
sphinx-build copy, dump  0:04:06.186373

Describe the solution you'd like
It would be ideal if Sphinx provided a means to set this value manually, e.g. as an argument for sphinx-build.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading sphinx/util/parallel.py around lines 137-150, then trace the sphinx-build argument and configuration entry points for parallel processing. Done means users can set maxbatch manually and that value controls the parallel read/write chunk size; compare the reported sphinx-build phase timings to confirm the setting is applied.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, performance
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.