allure-framework / allure-framework/allure-python
Categorize passed tests
- Langage dominant
- Python
- Étoiles
- 814
- Forks
- 260
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
[//]: # (
. Note: for support questions, please use Stackoverflow or Gitter**.
. This repository's issues are reserved for feature requests and bug reports.
.
. In case of any problems with Allure Jenkins plugin** please use the following repository
. to create an issue: https://github.com/jenkinsci/allure-plugin/issues
.
. Make sure you have a clear name for your issue. The name should start with a capital
. letter and no dot is required in the end of the sentence. An example of good issue names:
.
. - The report is broken in IE11
. - Add an ability to disable default plugins
. - Support emoji in test descriptions
)
#### I'm submitting a ...
- [ ] bug report
- [x] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.
#### What is the current behavior?
My team relies heavily on custom categorization of test results to make reviewing reports easier for our development teams. Currently, you cannot categorize "Passed" tests as status details are only included for broken or failed tests that have an exception.
Closes https://github.com/allure-framework/allure-python/issues/580
#### If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem
#### What is the expected behavior?
We would like to be able to also categorize passed tests without having to mark those tests as failed in order to get a status details attached to the result
#### What is the motivation / use case for changing the behavior?
We have tagging processes in place for our team to flag tests with known product issues and known test issues. Sometimes these known issues are the result of a flaky test or product issue that does not reproduce all of the time.
- It is important for test reviewers to know when a test fails with a known issue so they can focus on any new failures instead.
- It's also important for test developers to known when a passing test has a known issue because it isn't always communicated back to us when a bug gets fixed (ie we need to remove the test tag)
To accomplish this we currently have to group scenarios with known issues into separate categories -- "failed" or "passed" (like below). However this means we have to mark both sets of scenarios as "failed" so that the exception is included in the resulting report allowing us to categorize the results. This is less than ideal because when reviewing the history of a flaky issue, it's not clear how frequently the test actually passes / fails without clicking into each result to view the category

#### Suggested change
**One lightweight solution would be to also consider the `error_message` when getting a scenarios status details**. This would allow test developers to attach errors messages to other tests that did not fail with an exception. The cool thing about this is the test developer could also include skip reasons for skipped tests by simply setting the skip reason as the error message
```
# attach message to first step for passed test with known issue
if scenario.status == Status.passed and test_has_known_issue:
list(scenario.all_steps)[0].error_message = "Test passed with known issues"
```

#### Please tell us about your environment:
Allure version: 2.15.0
Test framework: behave=1.2.6
Allure adaptor: allure-behave==2.8.24
#### Other information
[//]: # (
. e.g. detailed explanation, stacktraces, related issues, suggestions
. how to fix, links for us to have more context, eg. Stackoverflow, Gitter etc
)
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par retracer la manière dont l’adaptateur allure-behave représente les détails de l’état du scénario et la manière dont error_message est géré pour les tests réussis. Vérifiez le comportement actuel avec Allure 2.15.0 et behave 1.2.6, puis définissez l’achèvement comme la possibilité de catégoriser les scénarios réussis comportant un message d’erreur sans les marquer comme échoués, tout en préservant le comportement existant des tests échoués.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python
- Domaine
- testing
- Type d'issue
- Fonctionnalité
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- Plutôt claire
- Accessibilité débutants
- 30/100