Publish C-level coverage of the CPython test suite

Abierto
#94,759 0 comentarios 4 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
5/5
Tiempo estimado
Más de una semana
Aptitud para principiantes
25/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Estancado
Stack tecnológico
github-actions, python
Área
ci-cd, testing-qa

Línea de trabajo

Empieza revisando el workflow existente .github/workflows/coverage.yml y la discusión enlazada sobre el uso de clang/llvm-cov. El MVP propuesto es un informe nocturno de cobertura de CPython a nivel de C publicado mediante GitHub Pages; revisa el PR gh-94760 para ver el trabajo que ya está en curso y lo que falta.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

tests type-feature

Feature or enhancement

Automatically publish C-level coverage of the CPython test suite.

Pitch

Recent issues, such as the problems with frame.setlineno, have revealed that the C-level coverage of the test suite isn't perfect. While source-level coverage isn't the only metric of a good test suite, paying closer attention to it should help us have more confidence when making changes to the CPython core.

Proposed design for MVP

  • Publish results nightly: This cadence should be enough for most cases. While various tools like codecov.io can run on every pull request, coverage takes a long time to run, and IME, there is enough noise in the results that developers (especially new to the project) would spend a lot of time "chasing" coverage.
  • Use clang/llvm-cov rather than gcc/gcov: As @erlend-aasland pointed out the LLVM toolchain coverage provides branch counts and coverage of subexpressions which seems pretty useful. There is a risk that this isn't the same tooling as most Linux developers would use, and not what make coverage-report does.
  • Publish to Github Pages: The easiest thing seems to be to publish to Github Pages in a new repository in the python org for this purpose. This keeps all of the credential management within Github.

Out of scope

Creating diffs between coverage runs seems like it would be useful, but I haven't found a good tool for that yet.

Python-level coverage reports would likewise be useful, but should be a follow-on to this.

Previous discussion

@brettcannon had set up coverage reports in the past, but they have been broken for a couple of releases. This aims to be simpler than that approach by using a pure Github stack (not using codecov.io) and would just run nightly, not as part of every pull request.

More recent discussion began here: https://github.com/faster-cpython/ideas/issues/426

  • PR: gh-94760
Lenguaje dominante
Python
Estrellas
77.2k
Forks
36k
Merge medio
1 d 9 h
PR fusionados (30 d)
558

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de python/cpython

Todos los issues de python/cpython

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.