sphinx-doc / sphinx-doc/sphinx
Allow setting maxbatch to adjust chunk size for parallel read/write
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8k
- Forks
- 2.6k
- PR merge metrics
- No merged PRs in 30d
Description
Is your feature request related to a problem? Please describe.
I'm currently working on a project with a large documentation (~22k files).
We noticed that reading in parallel is particularly slow:
sphinx-build init 0:00:47.712593
sphinx-build read 0:09:39.427063
sphinx-build write 0:03:55.736320
sphinx-build copy, dump 0:04:10.119199
I managed to track this issue down to the calculation of the chunk size for parallel processing:
https://github.com/sphinx-doc/sphinx/blob/cc314f13e8a98393ab018d83d8957a724a6f338a/sphinx/util/parallel.py#L137-L150
When setting maxbatch from 10 to 1000 the read performance improves significantly:
sphinx-build init 0:00:45.914173
sphinx-build read 0:00:50.814347
sphinx-build write 0:03:36.089072
sphinx-build copy, dump 0:04:06.186373
Describe the solution you'd like
It would be ideal if Sphinx provided a means to set this value manually, e.g. as an argument for sphinx-build.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading sphinx/util/parallel.py around lines 137-150, then trace the sphinx-build argument and configuration entry points for parallel processing. Done means users can set maxbatch manually and that value controls the parallel read/write chunk size; compare the reported sphinx-build phase timings to confirm the setting is applied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, performance
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100