allure-framework / allure-framework/allure2

Incorrect Success rate Calculation deducting SkippedScenario

Open
#2,685 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
5.5k
Forks
782
Avg merge
2d 33m
Merged PRs (30d)
16

Description

### Describe the Bug

Hello
it seems allure calculates the success rate = (SucceededScenario / (TotalScenario -SkippedScenario))
As a result we get different values than what is calculated in Azure devops & Allure report
( there is a sligth different in number of scenario counted by Azure & Allure because the QA used the same scenario name several times in one feature)

Example:
![image](https://github.com/user-attachments/assets/1e89377e-c51b-484a-a7c7-10e85e82502c)
![image](https://github.com/user-attachments/assets/faad8d21-98f8-42e4-a1ab-fcba794b152b)

The tooltip has the right success rate calculated but not the main graph.
![image](https://github.com/user-attachments/assets/bbfefc28-7795-400d-993e-555c8965a435)

the same issue has been noticed using allure report & Jenkins pipelines

in our context, When test are executed, if an error is found in one scenario it skipped the next following scenarios of the given feature;
it has been activated with

cucumberOpts: [
failFast: true
}

### Steps to Reproduce

Run several cucumber feature tests containing sevaral scenario that succeed, failed and skipped.
it seems allure calculates the success rate = 'SucceededScenario / (TotalScenario -SkippedScenario) )
SkiipedScenario shouldn't be deducted from the total number of scenario

### Expected Behaviour

success rate = (SucceededScenario / TotalScenario)
it it doesn't fit everybody an option should be considered to take into account SkippedScenario in the calculation or not.

### Screenshots or Additional Context

this issue has been discovered while using https://marketplace.visualstudio.com/items?itemName=qameta.allure-azure-pipelines&ssr=false#overview

### What Language are you using?

TypeScript

### What Framework/Allure Integration you are using?

wdio/allure-reporter": "8.24.0 / allure-commandline": "^2.29.0 /

### What version of Allure Integration you are using?

2.29.0

### What version of Allure Report you are using?

2.29.0

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start with the main report graph's success-rate calculation and compare it with the tooltip calculation; the issue says the tooltip is correct while the graph deducts skipped scenarios. Reproduce with Cucumber features containing succeeded, failed, and skipped scenarios, then verify that the graph uses succeeded scenarios divided by total scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
analytics, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.