sphinx-doc / sphinx-doc/sphinx

[tracker] [tests] Python test suite reliability: striving for non-flaky, parallelizable, random-orderable tests.

Open
#12,191 3 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

help wanted python type:task type:tests
Dominant language
Python
Stars
8k
Forks
2.6k
PR merge metrics
No merged PRs in 30d

Description

This is an umbrella issue to track progress making the Sphinx test suite reliable. That means:

  • No flaky tests.
  • Tests can run in parallel.
  • Tests are independent of each other.

Regarding the last item: test independence is important to ensure that each test is checking the behaviours it claims to, and does not inadvertently rely on the side-effects of some other test. Similarly, a test must not begin failing if any other test is run before it -- if it does, that could mean (but does not definitely mean) that the application's behaviour itself may vary based on the order in which code is evaluated.

It might seem like there are a large number of items here! However, please bear in mind that we have a total of more than 2000 individual pytest test cases that run.

Test flakiness

Test parallelization

Test independence

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the pytest suite and the listed cases under Test flakiness, Test parallelization, and Test independence, such as test_numfig_disabled_warn and tests.test_domains.test_domain_cpp. Run the relevant tests in isolation, parallel, and random order to identify failures. Done means the affected tests are reliable, independent, and parallelizable, with another flaky case tracked if found.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.