Publish C-level coverage of the CPython test suite

まず既存の .github/workflows/coverage.yml workflow と、clang/llvm-cov の使用に関するリンク先の議論を確認してください。提案されている MVP は、GitHub Pages で公開する夜間の C レベル CPython coverage レポートです。すでに進行中の作業と残っている作業を確認するため、PR gh-94760 を確認してください。

Feature or enhancement

Automatically publish C-level coverage of the CPython test suite.

Pitch

Recent issues, such as the problems with frame.setlineno, have revealed that the C-level coverage of the test suite isn't perfect. While source-level coverage isn't the only metric of a good test suite, paying closer attention to it should help us have more confidence when making changes to the CPython core.

Proposed design for MVP

• Publish results nightly: This cadence should be enough for most cases. While various tools like codecov.io can run on every pull request, coverage takes a long time to run, and IME, there is enough noise in the results that developers (especially new to the project) would spend a lot of time "chasing" coverage.
• Use clang/llvm-cov rather than gcc/gcov: As @erlend-aasland pointed out the LLVM toolchain coverage provides branch counts and coverage of subexpressions which seems pretty useful. There is a risk that this isn't the same tooling as most Linux developers would use, and not what make coverage-report does.
• Publish to Github Pages: The easiest thing seems to be to publish to Github Pages in a new repository in the python org for this purpose. This keeps all of the credential management within Github.

Out of scope

Creating diffs between coverage runs seems like it would be useful, but I haven't found a good tool for that yet.

Python-level coverage reports would likewise be useful, but should be a follow-on to this.

Previous discussion

@brettcannon had set up coverage reports in the past, but they have been broken for a couple of releases. This aims to be simpler than that approach by using a pure Github stack (not using codecov.io) and would just run nightly, not as part of every pull request.

More recent discussion began here: https://github.com/faster-cpython/ideas/issues/426

• PR: gh-94760