sphinx-doc / sphinx-doc/sphinx
[tracker] [tests] Python test suite reliability: striving for non-flaky, parallelizable, random-orderable tests.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8k
- Forks
- 2.6k
- PR merge metrics
- No merged PRs in 30d
Description
This is an umbrella issue to track progress making the Sphinx test suite reliable. That means:
- No flaky tests.
- Tests can run in parallel.
- Tests are independent of each other.
Regarding the last item: test independence is important to ensure that each test is checking the behaviours it claims to, and does not inadvertently rely on the side-effects of some other test. Similarly, a test must not begin failing if any other test is run before it -- if it does, that could mean (but does not definitely mean) that the application's behaviour itself may vary based on the order in which code is evaluated.
It might seem like there are a large number of items here! However, please bear in mind that we have a total of more than 2000 individual pytest test cases that run.
Test flakiness
-
test_build_linkcheck.test_connect_to_selfsigned_fails(pr: #12166) -
test_build_linkcheck.test_defaults(pr: #12166) -
test_build_linkcheck.test_linkcheck_exclude_documents(pr: #12189) -
test_numfig_disabled_warn(pr: #14107) -
test_n_slowest_value(pr: #14110) - ... report another?
Test parallelization
-
tests.test_builders.test_build_linkcheck(pr: #12126) - ... report another?
Test independence
-
tests.test_builders.test_build_text.py(pr: #12868) -
tests.test_domains.test_domain_cpp(pr: #12187) -
tests.test_builders.test_build_html(pr: #12118) -
tests.test_builders.test_build_html_download(pr: #12119) - ... report another?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the pytest suite and the listed cases under Test flakiness, Test parallelization, and Test independence, such as test_numfig_disabled_warn and tests.test_domains.test_domain_cpp. Run the relevant tests in isolation, parallel, and random order to identify failures. Done means the affected tests are reliable, independent, and parallelizable, with another flaky case tracked if found.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100