Separate doctests into a separate Nox session and GitHub workflow?
Open
Nobody has claimed this yet.
documentation infrastructure
nox
priority: low
- Dominant language
- Python
- Stars
- 711
- Forks
- 375
- Avg merge
- 3h 2m
- Merged PRs (30d)
- 8
Description
I'm wondering if it would be worthwhile to have a dedicated Nox session and GitHub workflow to check that doctests are working. At the moment I think it would be beneficial to do this, but it'd be worth discussing this further before attempting to implement this.
Advantages:
- We'd be more likely to notice if the doctests are not being run (see #2834)
- We could more reliably ensure that doctests are not counted when checking code coverage (so that we can identify lines of code that the test suite itself does not cover)
- We could shorten the time it takes to run the test suite. I'm not sure if the improvement will be negligible or noticeable, but I'd guess maybe a few seconds?
- We could have somewhat more computationally intensive examples without lengthening the overall time it takes tests to run.
- With
pytest-doctestplusenabled, we could have a Nox session that would check and optionally autofix docstrings. When doctests fail, we could print out the command to run to perform autofixes. - We could more easily isolate problems with doctests.
Disadvantages:
- When running Nox sessions locally, we'd have to run more than one to run test.
- We'd length the list of checks that get run in each PR.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin by reviewing the existing Nox sessions and GitHub workflows, then inspect issue #2834 to understand the current doctest coverage gap. Compare the trade-offs in this proposal, including pytest-doctestplus autofixes and coverage isolation; done requires an agreed design before implementation can be scoped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100