allure-framework / allure-framework/allure3
Allure Quality Gate doesn't work with the config file anymore
- Dominant language
- HTML
- Stars
- 401
- Forks
- 58
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 34
Description
### What happened?
I am using the allure quality-gate command to run my quality gate rules against my tests results with the following command:
`npx allure quality-gate ./allure-results --config ../../libs/testlib/testlib/allure_report/allurerc.mjs`
or
`npx allure quality-gate ./allure-results -c ../../libs/testlib/testlib/allure_report/allurerc.mjs`
i have a allure-results folder and no allure-report folder
my allurerc.mjs contains the config shown in the official documentation:
```qualityGate: {
rules: [
{
// use id field to make ruleset's rules easier to identify
id: "fast-failing ruleset",
maxFailures: 1,
fastFail: true,
},
{
id: "non-fast-failing ruleset",
successRate: 0.95,
},
],
},
```
it generates a allure-report folder containing one file 'test-results.json'. This file only contains this:
```{
"byId": {
"mytestid1": {
"id": "mytestid1",
"name": "my test name",
"duration": 19562,
"status": "failed"
},
"mytestid2": {
"id": "mytestid2",
"name": "my test name",
"duration": 19321,
"status": "failed"
},
}
}
```
i have 2 tests and both are KO, so both rules should fail but nothing is printed to the console.
However if i run the quality gate with the inline rule arguments, i get this:
```
npx allure quality-gate ./allure-results --max-failures 1
Quality Gate failed with following issues:
⨯ The number of failed tests 2 exceeds the allowed threshold value 1 maxFailures
1 quality gate rules have been failed.
```
if i run `npx allure quality-gate ./allure-results` i get an error message "Quality gate is not configured!
Add qualityGate to the config or consult help to know, how to use the command with command-line arguments"
and if i run `npx allure generate ./allure-results --config ../../libs/testlib/testlib/allure_report/allurerc.mjs` to generate a allure-report folder and the run `npx allure quality-gate ./allure-results --config ../../libs/testlib/testlib/allure_report/allurerc.mjs`, nothing is printed to the console and the same "test-results.json" file i mentioned is created inside my existing allure-report folder
So my guess is that the config file is recognized, but the rules defined in the config file aren't used
For reference, this was working properly in allure 3.14.0
### How can we reproduce it?
Using the config from the official documentation in my allurerc.mjs config file:
```
import { defineConfig } from "allure";
export default defineConfig({
qualityGate: {
rules: [
{
// use id field to make ruleset's rules easier to identify
id: "fast-failing ruleset",
maxFailures: 5,
fastFail: true,
},
{
id: "non-fast-failing ruleset",
successRate: 0.95,
},
],
},
});
```
Run 2 tests that are KO (i am using playwright typescript with the allure-playwright reporter) > this should generate the allure-results folder
Run npx allure quality-gate ./allure-results -c
The command runs and shows nothing in the terminal
a allure-report folder was created and contains 1 single file test-results.json
### What did you expect?
The command runs and shows the results of the quality gate defined in the config file against the tests results from the allure-results folder
### Environment
- Allure 3 version: 3.16.0
- Node.js version: v22.11.0
- OS: Windows
- @playwright/test": "1.59.1",
### Additional context
I need the console print to work with a config file because i want to declare custom rules, and i cant run my tests directly using allure run because i have some data enrichment that needs to run between my test run and my allure report generation
Contributor guide
Assessment
This issue has not been assessed yet.