allure-framework / allure-framework/allure3
Fast-fail Quality Gate false positive when framework-level retries are enabled
- Dominant language
- HTML
- Stars
- 401
- Forks
- 58
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
Currently, the fast fail quality gate feature doesn't work well in `allure run` with retries at the test framework level.
There are two places where the quality gate rules are applied during the `allure run` command:
1. In real time, right after newly available test results are passed into the storage. Here, the rules are only applied to the new test results to check if we should terminate the test run.
2. After the test run is finished. This is where the actual quality gate check is performed.
Since at (2) we're dealing with the most recent attempts only, this check always works as expected.
In contrast, at (2), the quality gate processes test results one by one without any prior knowledge of future attempts of the same test. To illustrate the unexpected effects this may lead to, let's define the following configuration:
```js
export default defineConfig({
plugins: {
log: {},
},
qualityGate: {
rules: [{
maxFailures: 1,
fastFail: true,
}],
},
});
```
Here, we've defined a fast-fail quality gate with a maximum failed tests threshold of 1. Our expectation is that the quality gate check fails if and only if we receive more than one failed test after the test run is complete. The `log` plugin will help us observe if the fast-fail behavior terminates the test run.
### Effect 1: fast-fail triggers because of multiple failed attempts of the same test
Here is a test (in Playwright) to observe the effect:
```js
import { expect, test } from "@playwright/test";
test.describe("foo", () => {
test.describe.configure({ retries: 2 });
test("Always fails", () => {
expect(true).toEqual(false);
});
test("Always passes", () => {});
});
```
We expect two failed attempts of `Always fails` to be counted as a single failure, which doesn't exceed `maxFailures`. Consequently, the fast-fail behavior is not triggered, and the `Always passes` test gets executed.
However, the actual quality gate check is positive, the execution is terminated, and the `Always passes` test is skipped:
```
...
yarn run playwright test ./test/retries.spec.ts --project chromium
process terminated (8176ms)
retries.spec.ts
⨯ Always fails 8ms
Error: expect(received).toEqual(expected) // deep equality
Expected: false
Received: true
⨯ expect.toEqual 1ms
Total tests: 1
Tests: 1 failed
Duration: 0.008s
Quality Gate failed with following issues:
⨯ The number of failed tests 2 exceeds the allowed threshold value 1 maxFailures
1 quality gate rules have been failed.
exit code 1 (8211ms)
```
If we set `fastFail` to `false`, the quality gate works as expected:
```
yarn run playwright test ./test/retries.spec.ts --project chromium
process finished with code 1 (9724ms)
retries.spec.ts
⨯ Always fails 4ms
Error: expect(received).toEqual(expected) // deep equality
Expected: false
Received: true
⨯ expect.toEqual 1ms
✓ Always passes 2ms
Total tests: 2
Tests: 1 passed | 1 failed
Duration: 0.006s
exit code 0 (9764ms)
```
### Effect 2: fast-fail triggers because tests passed after a retry are still counted as failed
The test:
```js
import { expect, test } from "@playwright/test";
test.describe("foo", () => {
test.describe.configure({ retries: 2 });
test("Passes on the second attempt", ({ page }, testInfo) => {
expect(testInfo.retry).toEqual(1);
});
});
test.describe("bar", () => {
test("Always fails", () => {
expect(true).toEqual(false);
});
test("Always passes", () => {});
});
```
> [!NOTE]:
> We've added an extra `Always passes` test to force Playwright to run longer, so the result of `Always fails` is checked in real-time, where the fast-fail behavior can be triggered.
We expect the quality gate check to pass because there is only one failed test at the end of the run. In reality, the check fails, and the test run is terminated, skipping the `Always passes` test execution:
```
yarn run playwright test ./test/retries.spec.ts --project chromium
process terminated (8158ms)
retries.spec.ts
✓ Passes on the second attempt 142ms
⨯ Always fails 1ms
Error: expect(received).toEqual(expected) // deep equality
Expected: false
Received: true
⨯ expect.toEqual 1ms
Total tests: 2
Tests: 1 passed | 1 failed
Duration: 0.143s
Quality Gate failed with following issues:
⨯ The number of failed tests 2 exceeds the allowed threshold value 1 maxFailures
1 quality gate rules have been failed.
```
If we disable `fastFail`, the quality gate works as expected:
```
yarn run playwright test ./test/retries.spec.ts --project chromium
process finished with code 1 (9198ms)
retries.spec.ts
⨯ Always fails 1ms
Error: expect(received).toEqual(expected) // deep equality
Expected: false
Received: true
⨯ expect.toEqual 1ms
✓ Passes on the second attempt 136ms
✓ Always passes 2ms
Total tests: 3
Tests: 2 passed | 1 failed
Duration: 0.139s
exit code 0 (9243ms)
```
### Effect 3: fast-fail terminates the run that would've otherwise made the test pass on one of the subsequent attempts
Here is the test:
```js
import { expect, test } from "@playwright/test";
test("Always fails", () => {
expect(true).toEqual(false);
});
test.describe("foo", () => {
test.describe.configure({ retries: 2 });
test("Passes on the second attempt", ({ page }, testInfo) => {
expect(testInfo.retry).toEqual(1);
});
test("Passes after 1s", async () => {
await new Promise((resolve) => setTimeout(resolve, 1000));
});
});
```
We need an extra test to be scheduled after the `Passes on the second attempt` to make Playwright run longer, allowing us to reliably observe the termination due to the fast-fail behavior.
The expectation is that the quality gate passes because we only have one failed test at the end of the run. In reality, the quality gate check fails after the first failed attempt of `Passes on the second attempt`. The test run is terminated, and the second attempt, which would've been successful, is never made.
```
yarn run playwright test ./test/retries.spec.ts --project chromium
process terminated (7766ms)
retries.spec.ts
⨯ Always fails 4ms
Error: expect(received).toEqual(expected) // deep equality
Expected: false
Received: true
⨯ expect.toEqual 1ms
⨯ Passes on the second attempt 276ms
Error: expect(received).toEqual(expected) // deep equality
Expected: 1
Received: 0
⨯ expect.toEqual 1ms
Total tests: 2
Tests: 2 failed
Duration: 0.28s
Quality Gate failed with following issues:
⨯ The number of failed tests 2 exceeds the allowed threshold value 1 maxFailures
1 quality gate rules have been failed.
exit code 1 (7807ms)
```
If fastFail is disabled, the quality gate works as expected:
```
yarn run playwright test ./test/retries.spec.ts --project chromium
process finished with code 1 (7435ms)
retries.spec.ts
✓ Passes on the second attempt 147ms
✓ Always passes
⨯ Always fails 1ms
Error: expect(received).toEqual(expected) // deep equality
Expected: false
Received: true
⨯ expect.toEqual 1ms
Total tests: 3
Tests: 2 passed | 1 failed
Duration: 0.148s
exit code 0 (7484ms)
```
### Solution
To solve all three problems, we should implement a set of three measures:
1. Count unique `historyId` values instead of `id`.
2. Take `passed` attempts into account.
3. Count final attempts only.
Measures 1 and 2 are straightforward.
Counting final attempts only is currently impossible, as this would require additional information to be associated with test results (e.g., a label/property like `finalAttempt`). That, in turn, requires support in all integrations with test frameworks and retry libraries, which is time-consuming and, in some cases, may not be possible due to the frameworks' API limitations.
### Recommendations
Given that we implement measures 1 and 2 (see above), the current recommendations for those who want to use `maxFailures` with `fastFail` and have retries at the test framework level are:
1. Make sure that if a test fails, its retries are executed by the same worker before any other test. Most test frameworks behave that way out of the box.
2. Make sure `maxFailures` is greater than the total number of parallel workers. If you want maxFailure to be less, add an extra rule with the value you need and `fastFail` disabled:
```js
export default defineConfig({
// ...
qualityGate: {
rules: [
{
maxFailures: 10,
fastFail: true,
},
{ maxFailures: 2 },
],
},
});
```
Contributor guide
Research direction
Start at the quality-gate processing used by the `allure run` command and trace the real-time check that consumes results one by one. Use the Playwright retry scenarios in the issue to verify that unique `historyId` values and passed attempts are handled correctly, and add coverage showing that fast-fail no longer triggers for duplicate or later-passing attempts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, playwright
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100