OpenAstronomy / OpenAstronomy/github-actions-workflows

FEAT: support running coverage measurements through `coverage.py` directly, instead of via `pytest-cov`

Open
#330 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
21
Forks
27
Avg merge
2d 9h
Merged PRs (30d)
1

Description

I discovered this while helping on #207, but the issue I want to report is actually quite orthogonal to that PR, so I'm opening this ticket separately.

In short, I would like to make pytest-cov a choice rather than a hard requirement for coverage reporting; this plugin is merely more than a middle man and provides no actual functionality on its own, so I would like to eliminate it from dependency graphs to reduce the "attack surface" (not exclusively, but also in the security sense) for dependees.

Running coverage directly isn't really challenging in itself, it should be as simple as the following substitution:

pytest ... --cov --cov-report=xml
coverage run  --parallel-mode -m pytest
coverage combine
coverage xml

Note that, by virtue of coverage being a hard dependency to pytest-cov, it would be perfectly backwards compatible to use it directly.

The difficulty lies in how to approach this; here's a summary of the design constraints we're facing.

  • tox.yml (via tox itself) can only append arguments to a command via matrix.pytest_flag, which tox redirects as posargs
  • in order to automatically form a coverage command dynamically, I need to be able to prepend coverage run -m before the pytest invoke
  • I would also need to somehow be able to add dynamically generated commands to invoke coverage combine or coverage xml ...

This just seems impossible to do in the most general case. My preferred solution would be to renounce doing this stuff dynamically and let users define their own commands (using pytest-cov or coverage directly, at their discretion); but of course that'd be a breaking change. I will try to come up with a backward compatible transition plan.

Originally posted by @neutrinoceros in https://github.com/OpenAstronomy/github-actions-workflows/issues/207#issuecomment-3523081930

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading tox.yml and the matrix.pytest_flag/posargs path described in the issue. Trace how coverage is currently invoked through pytest-cov and determine where dynamically adding coverage run, coverage combine, and coverage xml commands would fit. Done means direct coverage.py execution works while the backward-compatibility transition is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.