apache / apache/maven-surefire

[SUREFIRE-1691] surefire-junit47 reports results against incorrect test

Open
#2,115 1 comment 0 reactions 0 assignees View on GitHub
bug priority:major
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

**[M.P. Korstanje](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=mpkorstanje)** opened **[SUREFIRE-1691](https://issues.apache.org/jira/browse/SUREFIRE-1691?redirect=false)** and commented

This is a follow up on:
- https://issues.apache.org/jira/browse/SUREFIRE-1156
- https://github.com/cucumber/cucumber-jvm/issues/865

The latest reproducer can be found here:
- https://github.com/mpkorstanje/surefire-test/tree/07d0ee98ea424b38a3c548981a875dde1ef4f2c2

In short Cucumber has a hierarchy of tests that looks at follows:

```
Class annotated with @RunWith(Cucumber.class)
|- Feature File 1
| |- Scenario 1a
| |- Scenario 1b
|- Feature File 2
| |- Scenario 2a
| |- Scenario 2b
```

Surefire tries to group output in tests sets, either per class or per test suite. As Cucumber currently only emits test started/finished events for Scenarios sure fire can not group scenarios into a test set. They are instead grouped with the previous test set. However even after adding test started/finished events for features, sure fire groups all results in a single test set.

I believe this is caused by an error in `NonConcurrentRunListener.describesNewTestSet`.

```java
private boolean describesNewTestSet( Description description )
{
if ( currentTestSetDescription != null )
{
if ( null != description.getTestClass() )
{
return !description.getTestClass().equals( currentTestSetDescription.getTestClass() );
}
else if ( description.isSuite() )
{
return description.getChildren().equals( currentTestSetDescription.getChildren() );
}

return false;
}

return true;
}
```

The value of `description.getChildren().equals( currentTestSetDescription.getChildren() );` should be negated so so different test suits are properly recognized as test sets.

Fixing this locally works and can be reproduced by:

1. Negating the line and building sure fire locally
2. Building https://github.com/cucumber/cucumber-jvm/pull/1765 with `mvn clean install -DskipTests`
3. Running `mvn clean test -Pjunit47` on https://github.com/mpkorstanje/surefire-test

It does however does result in another unwanted result: A feature file with two scenarios is now reported as having 3 results. One for each scenario and one for the feature.

```
-------------------------------------------------------------------------------
Test set: Some Feature 1
-------------------------------------------------------------------------------
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 s - in Some_Feature_1
```

---

**Affects:** 3.0.0-M3

Contributor guide

Open the contributing guide

Research direction

Start with NonConcurrentRunListener.describesNewTestSet and reproduce the behavior using the linked surefire-test project and its junit47 profile. Verify that different feature suites become separate test sets while checking the reported result count for a feature with two scenarios. Done means correct grouping without an unwanted extra feature result.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.