allure-framework / allure-framework/allure-java

TestNG retries are marked as Skipped and the test itself is not marked as Flaky Test

Offen
#452 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Java
Sterne
376
Forks
240
Ø Merge
1 T. 7 Std.
Gemergte PRs (30 T.)
45

Beschreibung

[//]: # (
. Note: for support questions, please use Stackoverflow or Gitter**.
. This repository's issues are reserved for feature requests and bug reports.
.
. In case of any problems with Allure Jenkins plugin** please use the following repository
. to create an issue: https://github.com/jenkinsci/allure-plugin/issues
.
. Make sure you have a clear name for your issue. The name should start with a capital
. letter and no dot is required in the end of the sentence. An example of good issue names:
.
. - The report is broken in IE11
. - Add an ability to disable default plugins
. - Support emoji in test descriptions
)

#### I'm submitting a ...
- [X] bug report
- [ ] feature request
- [ ] support request => Please do not submit support request here, see note at the top of this template.

#### What is the current behavior?
Tests that failed and then success after re-run using TestNG retryAnalyzer marked as skipped and not showing the Flaky Test (bomb) icon.
![image](https://user-images.githubusercontent.com/29943195/84980868-669f1d00-b15d-11ea-83ed-065a5766d9dc.png)
![image](https://user-images.githubusercontent.com/29943195/84980908-80d8fb00-b15d-11ea-88e9-962d5f49bb3d.png)

#### If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem
The retry impl class

public class RetryAnalyzerImpl implements IRetryAnalyzer {
private int maxRetryCount = 3;
private int retryCount = 0;

public boolean retry(ITestResult result) {
if (retryCount < maxRetryCount) {
retryCount++;
return true;
}
return false;
}
}

The test method

@Test(retryAnalyzer= RetryAnalyzerImpl.class)
public void VerifyRetrievePublicaAndActiveCommunity(){
int i = ThreadLocalRandom.current().nextInt(0, 2);
System.out.println(i);
Assert.assertEquals(i,0);
}

#### What is the expected behavior?
The retries are marked as Failed correctly. And the tests are marked as Flaky.

#### What is the motivation / use case for changing the behavior?

#### Please tell us about your environment:

| Allure version | 2.13.3 |
| Test framework | testng@7.1.0 |
| Allure integration | allure-testng@2.13.3 |
| Generate report using | allure windows cli@2.13.3 |

#### Other information

[//]: # (
. e.g. detailed explanation, stacktraces, related issues, suggestions
. how to fix, links for us to have more context, eg. Stackoverflow, Gitter etc
)

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne damit, das Verhalten mit dem bereitgestellten RetryAnalyzerImpl und dem TestNG-Test unter Verwendung von allure-testng 2.13.3 zu reproduzieren. Verfolge anschließend, wie Wiederholungsversuche in der TestNG-Integration und der Berichtsausgabe klassifiziert werden. Fertig ist die Aufgabe, wenn fehlgeschlagene Wiederholungsversuche nicht als übersprungen gemeldet werden und der Test den Indikator Flaky Test anzeigt.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
testing
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
35/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.