allure-framework / allure-framework/allure-python

Feature request: Generate "unknown" test results during pytest collection (allure-pytest)

オープン
#821 コメント 1 件 リアクション 3 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
814
フォーク
260
PR マージ指標
30日以内にマージされた PR はありません

説明

[//]: # (
. Note: for support questions, please use Stackoverflow or Gitter**.
. This repository's issues are reserved for feature requests and bug reports.
.
. In case of any problems with Allure Jenkins plugin** please use the following repository
. to create an issue: https://github.com/jenkinsci/allure-plugin/issues
.
. Make sure you have a clear name for your issue. The name should start with a capital
. letter and no dot is required in the end of the sentence. An example of good issue names:
.
. - The report is broken in IE11
. - Add an ability to disable default plugins
. - Support emoji in test descriptions
)

#### I'm submitting a ...
- [ ] bug report
- [x] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.

#### What is the current behavior?
allure-pytest generates test results for each test after it finishes

#### If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

#### What is the expected behavior?
allure-pytest generates "unknown" test result for each test during collection time & replaces that test result if the test completes

#### What is the motivation / use case for changing the behavior?
Currently, on our CI, if the test step times out (because of GitHub Actions [`timeout-minutes`](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepstimeout-minutes)), tests that did not complete will be missing from the Allure Report.

We also distribute our tests across multiple GitHub Actions runners (to run in parallel), and if the set up of a runner fails, all of its tests will be missing from the Allure Report

#### Please tell us about your environment:

- Allure version: 2.29.0-1
- Test framework: pytest@7.4.0
- Allure adaptor: allure-pytest@2.13.5

#### Other information

We would be happy to open a PR to implement this, but we're not sure how to refactor/structure the code (currently, most of the logic to generate test reports is coupled to the pytest hooks)

There's two use cases here:
1. for normal test execution, generate unknown test results during collection time & replace them during execution time
2. for running tests with collect-only, generate unknown test results
- this is our use case, since we run pytest collection on one runner & use the results to provision multiple runners. If any of the test execution runners fail to set up, we would still like to show "unknown" results for all tests that were scheduled on that runner

We've actually implemented this already in a pytest plugin that imports from `allure-pytest`, but would love to upstream this (if that's something you'd want) so we can de-duplicate this code & so that our plugin doesn't break if allure-pytest code internals are modified
- Here's the pytest plugin that generates test results during collection time, if `--allure-collection-dir` is passed: https://github.com/canonical/data-platform-workflows/blob/v13.3.4/python/pytest_plugins/allure_pytest_collection_report/allure_pytest_collection_report/_plugin.py
- And here's a python script that combines the "default results" (generated during collection time) with the actual results: https://github.com/canonical/data-platform-workflows/blob/v13.3.4/python/cli/data_platform_workflows_cli/allure_add_default_for_missing_results.py
- full context of how we integrated this into our CI: https://github.com/canonical/data-platform-workflows/pull/183

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

まず、リンクされている allure_pytest_collection_report/_plugin.py とスクリプト allure_add_default_for_missing_results.py を確認し、次に allure-pytest にある既存の pytest フックを追跡します。コレクションによって unknown 結果が作成され、完了したテストによってそれらが置き換えられ、collect-only の実行で一度も実行されないテストの結果が保持されれば完了です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
testing-qa
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。