allure-framework / allure-framework/allure-python

Feature request: Generate "unknown" test results during pytest collection (allure-pytest)

Abierto
#821 1 comentario 3 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
814
Forks
260
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

[//]: # (
. Note: for support questions, please use Stackoverflow or Gitter**.
. This repository's issues are reserved for feature requests and bug reports.
.
. In case of any problems with Allure Jenkins plugin** please use the following repository
. to create an issue: https://github.com/jenkinsci/allure-plugin/issues
.
. Make sure you have a clear name for your issue. The name should start with a capital
. letter and no dot is required in the end of the sentence. An example of good issue names:
.
. - The report is broken in IE11
. - Add an ability to disable default plugins
. - Support emoji in test descriptions
)

#### I'm submitting a ...
- [ ] bug report
- [x] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.

#### What is the current behavior?
allure-pytest generates test results for each test after it finishes

#### If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

#### What is the expected behavior?
allure-pytest generates "unknown" test result for each test during collection time & replaces that test result if the test completes

#### What is the motivation / use case for changing the behavior?
Currently, on our CI, if the test step times out (because of GitHub Actions [`timeout-minutes`](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepstimeout-minutes)), tests that did not complete will be missing from the Allure Report.

We also distribute our tests across multiple GitHub Actions runners (to run in parallel), and if the set up of a runner fails, all of its tests will be missing from the Allure Report

#### Please tell us about your environment:

- Allure version: 2.29.0-1
- Test framework: pytest@7.4.0
- Allure adaptor: allure-pytest@2.13.5

#### Other information

We would be happy to open a PR to implement this, but we're not sure how to refactor/structure the code (currently, most of the logic to generate test reports is coupled to the pytest hooks)

There's two use cases here:
1. for normal test execution, generate unknown test results during collection time & replace them during execution time
2. for running tests with collect-only, generate unknown test results
- this is our use case, since we run pytest collection on one runner & use the results to provision multiple runners. If any of the test execution runners fail to set up, we would still like to show "unknown" results for all tests that were scheduled on that runner

We've actually implemented this already in a pytest plugin that imports from `allure-pytest`, but would love to upstream this (if that's something you'd want) so we can de-duplicate this code & so that our plugin doesn't break if allure-pytest code internals are modified
- Here's the pytest plugin that generates test results during collection time, if `--allure-collection-dir` is passed: https://github.com/canonical/data-platform-workflows/blob/v13.3.4/python/pytest_plugins/allure_pytest_collection_report/allure_pytest_collection_report/_plugin.py
- And here's a python script that combines the "default results" (generated during collection time) with the actual results: https://github.com/canonical/data-platform-workflows/blob/v13.3.4/python/cli/data_platform_workflows_cli/allure_add_default_for_missing_results.py
- full context of how we integrated this into our CI: https://github.com/canonical/data-platform-workflows/pull/183

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza revisando allure_pytest_collection_report/_plugin.py y el script allure_add_default_for_missing_results.py; después, sigue los hooks de pytest existentes en allure-pytest. Se considera terminado cuando la recopilación crea resultados desconocidos, las pruebas completadas los reemplazan y las ejecuciones de collect-only conservan los resultados de las pruebas que nunca se ejecutan.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
testing-qa
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
35/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.