allure-framework / allure-framework/allure-python
Categorize passed tests
- Dominant language
- Python
- Stars
- 814
- Forks
- 260
- PR merge metrics
- No merged PRs in 30d
Description
[//]: # (
. Note: for support questions, please use Stackoverflow or Gitter**.
. This repository's issues are reserved for feature requests and bug reports.
.
. In case of any problems with Allure Jenkins plugin** please use the following repository
. to create an issue: https://github.com/jenkinsci/allure-plugin/issues
.
. Make sure you have a clear name for your issue. The name should start with a capital
. letter and no dot is required in the end of the sentence. An example of good issue names:
.
. - The report is broken in IE11
. - Add an ability to disable default plugins
. - Support emoji in test descriptions
)
#### I'm submitting a ...
- [ ] bug report
- [x] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.
#### What is the current behavior?
My team relies heavily on custom categorization of test results to make reviewing reports easier for our development teams. Currently, you cannot categorize "Passed" tests as status details are only included for broken or failed tests that have an exception.
Closes https://github.com/allure-framework/allure-python/issues/580
#### If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem
#### What is the expected behavior?
We would like to be able to also categorize passed tests without having to mark those tests as failed in order to get a status details attached to the result
#### What is the motivation / use case for changing the behavior?
We have tagging processes in place for our team to flag tests with known product issues and known test issues. Sometimes these known issues are the result of a flaky test or product issue that does not reproduce all of the time.
- It is important for test reviewers to know when a test fails with a known issue so they can focus on any new failures instead.
- It's also important for test developers to known when a passing test has a known issue because it isn't always communicated back to us when a bug gets fixed (ie we need to remove the test tag)
To accomplish this we currently have to group scenarios with known issues into separate categories -- "failed" or "passed" (like below). However this means we have to mark both sets of scenarios as "failed" so that the exception is included in the resulting report allowing us to categorize the results. This is less than ideal because when reviewing the history of a flaky issue, it's not clear how frequently the test actually passes / fails without clicking into each result to view the category

#### Suggested change
**One lightweight solution would be to also consider the `error_message` when getting a scenarios status details**. This would allow test developers to attach errors messages to other tests that did not fail with an exception. The cool thing about this is the test developer could also include skip reasons for skipped tests by simply setting the skip reason as the error message
```
# attach message to first step for passed test with known issue
if scenario.status == Status.passed and test_has_known_issue:
list(scenario.all_steps)[0].error_message = "Test passed with known issues"
```

#### Please tell us about your environment:
Allure version: 2.15.0
Test framework: behave=1.2.6
Allure adaptor: allure-behave==2.8.24
#### Other information
[//]: # (
. e.g. detailed explanation, stacktraces, related issues, suggestions
. how to fix, links for us to have more context, eg. Stackoverflow, Gitter etc
)
Contributor guide
Assessment
This issue has not been assessed yet.