allure-framework / allure-framework/allure2
Incorrect Success rate Calculation deducting SkippedScenario
- Dominant language
- HTML
- Stars
- 5.5k
- Forks
- 782
- Avg merge
- 2d 33m
- Merged PRs (30d)
- 16
Description
### Describe the Bug
Hello
it seems allure calculates the success rate = (SucceededScenario / (TotalScenario -SkippedScenario))
As a result we get different values than what is calculated in Azure devops & Allure report
( there is a sligth different in number of scenario counted by Azure & Allure because the QA used the same scenario name several times in one feature)
Example:


The tooltip has the right success rate calculated but not the main graph.

the same issue has been noticed using allure report & Jenkins pipelines
in our context, When test are executed, if an error is found in one scenario it skipped the next following scenarios of the given feature;
it has been activated with
cucumberOpts: [
failFast: true
}
### Steps to Reproduce
Run several cucumber feature tests containing sevaral scenario that succeed, failed and skipped.
it seems allure calculates the success rate = 'SucceededScenario / (TotalScenario -SkippedScenario) )
SkiipedScenario shouldn't be deducted from the total number of scenario
### Expected Behaviour
success rate = (SucceededScenario / TotalScenario)
it it doesn't fit everybody an option should be considered to take into account SkippedScenario in the calculation or not.
### Screenshots or Additional Context
this issue has been discovered while using https://marketplace.visualstudio.com/items?itemName=qameta.allure-azure-pipelines&ssr=false#overview
### What Language are you using?
TypeScript
### What Framework/Allure Integration you are using?
wdio/allure-reporter": "8.24.0 / allure-commandline": "^2.29.0 /
### What version of Allure Integration you are using?
2.29.0
### What version of Allure Report you are using?
2.29.0
### Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Contributor guide
Research direction
Start with the main report graph's success-rate calculation and compare it with the tooltip calculation; the issue says the tooltip is correct while the graph deducts skipped scenarios. Reproduce with Cucumber features containing succeeded, failed, and skipped scenarios, then verify that the graph uses succeeded scenarios divided by total scenarios.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- analytics, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100