Publish C-level coverage of the CPython test suite

Open
#94,759 0 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
github-actions, python
Domain
ci-cd, testing-qa

Research direction

Start by reviewing the existing .github/workflows/coverage.yml workflow and the linked discussion about using clang/llvm-cov. The proposed MVP is a nightly C-level CPython coverage report published through GitHub Pages; check PR gh-94760 for work already underway and what remains.

Written by the indexing model from the issue text.

Description

tests type-feature

Feature or enhancement

Automatically publish C-level coverage of the CPython test suite.

Pitch

Recent issues, such as the problems with frame.setlineno, have revealed that the C-level coverage of the test suite isn't perfect. While source-level coverage isn't the only metric of a good test suite, paying closer attention to it should help us have more confidence when making changes to the CPython core.

Proposed design for MVP

  • Publish results nightly: This cadence should be enough for most cases. While various tools like codecov.io can run on every pull request, coverage takes a long time to run, and IME, there is enough noise in the results that developers (especially new to the project) would spend a lot of time "chasing" coverage.
  • Use clang/llvm-cov rather than gcc/gcov: As @erlend-aasland pointed out the LLVM toolchain coverage provides branch counts and coverage of subexpressions which seems pretty useful. There is a risk that this isn't the same tooling as most Linux developers would use, and not what make coverage-report does.
  • Publish to Github Pages: The easiest thing seems to be to publish to Github Pages in a new repository in the python org for this purpose. This keeps all of the credential management within Github.

Out of scope

Creating diffs between coverage runs seems like it would be useful, but I haven't found a good tool for that yet.

Python-level coverage reports would likewise be useful, but should be a follow-on to this.

Previous discussion

@brettcannon had set up coverage reports in the past, but they have been broken for a couple of releases. This aims to be simpler than that approach by using a pure Github stack (not using codecov.io) and would just run nightly, not as part of every pull request.

More recent discussion began here: https://github.com/faster-cpython/ideas/issues/426

  • PR: gh-94760
Dominant language
Python
Stars
77.2k
Forks
36k
Avg merge
1d 9h
Merged PRs (30d)
558

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from python/cpython

All issues in python/cpython

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.