allure-framework / allure-framework/allure-python

Feature request: Generate "unknown" test results during pytest collection (allure-pytest)

Open
#821 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
814
Forks
260
PR merge metrics
No merged PRs in 30d

Description

[//]: # (
. Note: for support questions, please use Stackoverflow or Gitter**.
. This repository's issues are reserved for feature requests and bug reports.
.
. In case of any problems with Allure Jenkins plugin** please use the following repository
. to create an issue: https://github.com/jenkinsci/allure-plugin/issues
.
. Make sure you have a clear name for your issue. The name should start with a capital
. letter and no dot is required in the end of the sentence. An example of good issue names:
.
. - The report is broken in IE11
. - Add an ability to disable default plugins
. - Support emoji in test descriptions
)

#### I'm submitting a ...
- [ ] bug report
- [x] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.

#### What is the current behavior?
allure-pytest generates test results for each test after it finishes

#### If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

#### What is the expected behavior?
allure-pytest generates "unknown" test result for each test during collection time & replaces that test result if the test completes

#### What is the motivation / use case for changing the behavior?
Currently, on our CI, if the test step times out (because of GitHub Actions [`timeout-minutes`](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepstimeout-minutes)), tests that did not complete will be missing from the Allure Report.

We also distribute our tests across multiple GitHub Actions runners (to run in parallel), and if the set up of a runner fails, all of its tests will be missing from the Allure Report

#### Please tell us about your environment:

- Allure version: 2.29.0-1
- Test framework: pytest@7.4.0
- Allure adaptor: allure-pytest@2.13.5

#### Other information

We would be happy to open a PR to implement this, but we're not sure how to refactor/structure the code (currently, most of the logic to generate test reports is coupled to the pytest hooks)

There's two use cases here:
1. for normal test execution, generate unknown test results during collection time & replace them during execution time
2. for running tests with collect-only, generate unknown test results
- this is our use case, since we run pytest collection on one runner & use the results to provision multiple runners. If any of the test execution runners fail to set up, we would still like to show "unknown" results for all tests that were scheduled on that runner

We've actually implemented this already in a pytest plugin that imports from `allure-pytest`, but would love to upstream this (if that's something you'd want) so we can de-duplicate this code & so that our plugin doesn't break if allure-pytest code internals are modified
- Here's the pytest plugin that generates test results during collection time, if `--allure-collection-dir` is passed: https://github.com/canonical/data-platform-workflows/blob/v13.3.4/python/pytest_plugins/allure_pytest_collection_report/allure_pytest_collection_report/_plugin.py
- And here's a python script that combines the "default results" (generated during collection time) with the actual results: https://github.com/canonical/data-platform-workflows/blob/v13.3.4/python/cli/data_platform_workflows_cli/allure_add_default_for_missing_results.py
- full context of how we integrated this into our CI: https://github.com/canonical/data-platform-workflows/pull/183

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked allure_pytest_collection_report/_plugin.py and the script allure_add_default_for_missing_results.py, then trace the existing pytest hooks in allure-pytest. Done means collection creates unknown results, completed tests replace them, and collect-only runs retain results for tests that never execute.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.