allure-framework / allure-framework/allure3
Advanced flakiness detection
- Dominant language
- HTML
- Stars
- 401
- Forks
- 58
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
Allure 2 and Allure 3 after #157 imply two sources of flakiness:
1. Integrations (via the `TestResult.statusDetails.flaky` property).
2. History (by checking the latest 5 runs).
In reality, many users prefer marking tests as flaky based on something else, e.g.:
1. Retries.
2. Custom function of the test result (e.g., by an AI assistant).
3. External sources (e.g., labels on GitHub).
Some would like those sources to be checked in some particular order, where the next source is checked if the previous one doesn't mark the test as flaky. An example of such a sequence could be:
1. Explicit property.
2. History.
3. Retries.
4. Custom.
Others prefer having the ability to make a final decision (or overwrite a previously made one). For example, they would like to keep some tests from being marked as flaky in some circumstances (e.g., in case the status details indicate a network failure they are not interested in).
Parameters of flaky detection algorithms (such as the depth of the history to account for) may also be subject to configuration.
Once #157 is done, we should come up with a mechanism to configure all those things and implement it. Once that's done, most of the use cases related to flakiness detection will be covered.
### Open questions
1. Source priority (order): fixed vs configurable; if it's the latter, should explicit flaky be always top priority, or should it be subject to the common rules?
2. Scope: Should the same rules apply for the entire Allure 3 session or only for Allure Awesome?
3. Source selection feedback: Should we (and if yes, then how) indicate to users which source has marked a given test as flaky?
### Thoughts on implementation
We may define a flaky detection rule as a function that takes a test result (or some data from it) and returns one of three decisions:
1. `flaky`: The test is definitely flaky.
2. `not flaky`: The test is definitely not flaky.
3. `unknown`: The rule is uncertain if the test is flaky. It's up to the following rules to make the decision.
Then, we can make it configurable to return `not flaky` instead of `unknown` for built-in rules to make their decision final.
A configuration may look something like this:
```ts
{
flakyDetection: {
rules: [
/* a built-in rule with no configuration changes */
"explicit", // alternatively, we may assume it's always on and comes first; on the other hand, making it configurable allows things like ignoring all `flaky` properties of test results to enforce team-specific rules.
/* A new custom rule */
{
name: "Known issues", // to show in the report if the rule has set the test as flaky
description: "Issues we know about and don't want to affect flakiness",
/* the rule implementation; I don't think passing a TestResult instance is a good idea, but for the sake of simplicity, let it be here for now */
isFlaky: async (tr: TestResult) => KNOWN_ERRORS.includes(tr.message)
? "not flaky" // will mark as not flaky and stop
: "unknown", // go to the next rule in the sequence (if any)
},
/* A built-in rule with configuration changed (no implementation, a well-known id) */
{
id: "history",
depth: 7,
},
/* Another built-in rule with no configuration changes */
"retries",
/* Another custom rule */
{
name: "AI",
description: "Asks an AI assistant if a given test is flaky",
/* the rule implementation */
isFlaky: async (tr: TestResult) => await askAssistantIfTestIsFlaky(tr)
? "unknown" // go to the next rule in the sequence (there is none, so the test will be marked as not flaky)
: "flaky", // mark as flaky and stop
},
},
},
}
```
### Related issues
- allure-framework/allure2#2361
Contributor guide
Research direction
Start by reviewing #157 and the current flakiness behavior based on statusDetails.flaky and the latest five runs. Then resolve the open questions about rule priority, session scope, and source feedback before designing the configurable rules and parameters. Done means the agreed sources, custom decisions, and history depth are supported.
Written by the indexing model from the issue text.
Assessment
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100