allure-framework / allure-framework/allure3
Indicate tests that changed their status in a retry
- Dominant language
- HTML
- Stars
- 401
- Forks
- 58
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
### The context
Currently, the `retried` indicator is shown next to a test if it has at least one retry, regardless of the outcomes.
In Allure 2, such an indicator is shown only if the test has different outcomes across its attempts.
Both implementations have their own advantages:
- The current implementation allows assessing the impact the retrying policy has on the test base.
- The Allure 2 implementation allows assessing the efficiency of the retrying policy in terms of the number of instabilities it detects. It also keeps flaky tests on the radar better (we can distinguish between tests that passed on the first try and those that passed after a retry).
On the other hand, the current implementation has a disadvantage: it makes the report noisier in case retries are unconditionally enabled for all tests. Some users employ such a strategy to define flaky tests ([an example](https://github.com/allure-framework/allure3/issues/160#issuecomment-2996565479)).
### The proposal
1. We should highlight tests with different statuses across their attempts.
2. We should show the total number of such tests.
### Open questions
We may either rework the current `retried` indication and counter or add a new one.
I personally prefer the latter as it will allow a better understanding of the efficiency of the retrying policy. For example, we'll be able to further asses the efficiency by comparing the number of flipped tests with the total number of retried tests. However, the downside is that it may make the report noisier and more difficult to read.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by locating the current retried indication and counter in the report, then trace how attempt outcomes are represented; done means tests with differing statuses are highlighted and their total is shown, with the chosen treatment of the existing indication documented.
Written by the indexing model from the issue text.
Assessment
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100