allure-framework / allure-framework/allure3

Indicate tests that changed their status in a retry

Open
#224 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
401
Forks
58
Avg merge
2d 20h
Merged PRs (30d)
34

Description

### The context

Currently, the `retried` indicator is shown next to a test if it has at least one retry, regardless of the outcomes.

In Allure 2, such an indicator is shown only if the test has different outcomes across its attempts.

Both implementations have their own advantages:

- The current implementation allows assessing the impact the retrying policy has on the test base.
- The Allure 2 implementation allows assessing the efficiency of the retrying policy in terms of the number of instabilities it detects. It also keeps flaky tests on the radar better (we can distinguish between tests that passed on the first try and those that passed after a retry).

On the other hand, the current implementation has a disadvantage: it makes the report noisier in case retries are unconditionally enabled for all tests. Some users employ such a strategy to define flaky tests ([an example](https://github.com/allure-framework/allure3/issues/160#issuecomment-2996565479)).

### The proposal

1. We should highlight tests with different statuses across their attempts.
2. We should show the total number of such tests.

### Open questions

We may either rework the current `retried` indication and counter or add a new one.

I personally prefer the latter as it will allow a better understanding of the efficiency of the retrying policy. For example, we'll be able to further asses the efficiency by comparing the number of flipped tests with the total number of retried tests. However, the downside is that it may make the report noisier and more difficult to read.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by locating the current retried indication and counter in the report, then trace how attempt outcomes are represented; done means tests with differing statuses are highlighted and their total is shown, with the chosen treatment of the existing indication documented.

Written by the indexing model from the issue text.

Assessment

Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.