allure-framework / allure-framework/allure-python

Categorize passed tests

Offen
#649 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
task:new feature theme:behave
Vorherrschende Sprache
Python
Sterne
814
Forks
260
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

[//]: # (
. Note: for support questions, please use Stackoverflow or Gitter**.
. This repository's issues are reserved for feature requests and bug reports.
.
. In case of any problems with Allure Jenkins plugin** please use the following repository
. to create an issue: https://github.com/jenkinsci/allure-plugin/issues
.
. Make sure you have a clear name for your issue. The name should start with a capital
. letter and no dot is required in the end of the sentence. An example of good issue names:
.
. - The report is broken in IE11
. - Add an ability to disable default plugins
. - Support emoji in test descriptions
)

#### I'm submitting a ...
- [ ] bug report
- [x] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.

#### What is the current behavior?
My team relies heavily on custom categorization of test results to make reviewing reports easier for our development teams. Currently, you cannot categorize "Passed" tests as status details are only included for broken or failed tests that have an exception.

Closes https://github.com/allure-framework/allure-python/issues/580

#### If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

#### What is the expected behavior?
We would like to be able to also categorize passed tests without having to mark those tests as failed in order to get a status details attached to the result

#### What is the motivation / use case for changing the behavior?
We have tagging processes in place for our team to flag tests with known product issues and known test issues. Sometimes these known issues are the result of a flaky test or product issue that does not reproduce all of the time.
- It is important for test reviewers to know when a test fails with a known issue so they can focus on any new failures instead.
- It's also important for test developers to known when a passing test has a known issue because it isn't always communicated back to us when a bug gets fixed (ie we need to remove the test tag)

To accomplish this we currently have to group scenarios with known issues into separate categories -- "failed" or "passed" (like below). However this means we have to mark both sets of scenarios as "failed" so that the exception is included in the resulting report allowing us to categorize the results. This is less than ideal because when reviewing the history of a flaky issue, it's not clear how frequently the test actually passes / fails without clicking into each result to view the category

![image](https://user-images.githubusercontent.com/10168933/152859806-dafd2202-c048-464a-8c36-60a2d72b4ed0.png)

#### Suggested change

**One lightweight solution would be to also consider the `error_message` when getting a scenarios status details**. This would allow test developers to attach errors messages to other tests that did not fail with an exception. The cool thing about this is the test developer could also include skip reasons for skipped tests by simply setting the skip reason as the error message

```
# attach message to first step for passed test with known issue
if scenario.status == Status.passed and test_has_known_issue:
list(scenario.all_steps)[0].error_message = "Test passed with known issues"
```

![image](https://user-images.githubusercontent.com/10168933/152859622-c24010d1-80e8-4f57-8c4d-34fd622f8aca.png)

#### Please tell us about your environment:

Allure version: 2.15.0
Test framework: behave=1.2.6
Allure adaptor: allure-behave==2.8.24

#### Other information

[//]: # (
. e.g. detailed explanation, stacktraces, related issues, suggestions
. how to fix, links for us to have more context, eg. Stackoverflow, Gitter etc
)

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne damit nachzuverfolgen, wie der Adapter allure-behave Details zum Szenariostatus darstellt und wie error_message für bestandene Tests behandelt wird. Überprüfe das aktuelle Verhalten anhand von Allure 2.15.0 und behave 1.2.6 und definiere den Abschluss so, dass bestandene Szenarien mit einer Fehlermeldung kategorisiert werden können, ohne sie als fehlgeschlagen zu markieren, wobei das bestehende Verhalten für fehlgeschlagene Tests erhalten bleibt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
testing
Issue-Typ
Feature
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
30/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.