openedx / openedx/openedx-platform

Enforce 100% coverage on the test modules themselves

Open
#32,659 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

code health
Dominant language
Python
Stars
8.2k
Forks
4.4k
Avg merge
6d 18h
Merged PRs (30d)
42

Description

More specifically: check for 100% coverage on modules named */tests/* or */test_*.py as a required PR check.

This is not a proposal to bring edx-platform as a whole to 100% coverage.

Details

Enforcing a coverage threshold on application code certainly comes with its pros and cons. But we can all agree that if a test case is added to edx-platform, running the test suite should run that test case, right?

This would prevent folks from adding code that they think will be run as unit tests, but due to an honest mistake, won't be. That happened here:

This would also shield us from tooling issues which might silently stop part of the test suite from running for some amount of time, potentially letting swaths of tests regress until it's noticed. This happened to the code in xmodule/ once (back when it was common/lib/xmodule/xmodule and had its own test settings), and we were lucky that the suite was easy to restore.

An implication of this, which I'm personally OK with, is that it would enforce that test helpers get removed if the unit tests that call them are removed. I'm sure there are some individual exceptions (eg, maybe codejail-related tests can only be run locally), but those should be possible to exempt as needed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the coverage configuration and required PR-check setup for Python paths matching */tests/* or */test_*.py, then review the three referenced pull requests for failure cases. Done means matching test modules are required to reach 100% coverage, with an explicit way to handle documented exceptions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.