apache / apache/maven-surefire
[BUG] TXT report shows "Tests run: 0" for outer class when all tests are in @Nested classes
- Dominant language
- Java
- Stars
- 461
- Forks
- 588
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 19
Description
## Summary
When all test methods in a JUnit 5 test class live inside `@Nested` inner classes (none at the outer level), the `.txt` surefire report for the outer class shows `Tests run: 0, Failures: 0, Errors: 0` — regardless of whether tests pass, fail, or error. The XML report for the same class correctly reflects the actual test count.
## Environment
- `maven-surefire-plugin`: **3.5.5**
- JUnit Jupiter: 5.11.4
- Java: 17
## Steps to reproduce
Minimal reproducer: https://github.com/shitikanth/surefire-nested-txt-report-reproducer
```java
class OuterTest {
@Nested
class GroupA {
@Test void test_one() { assertTrue(true); }
@Test void test_two() { assertTrue(true); }
}
@Nested
class GroupB {
@Test void test_three() { assertTrue(true); }
}
}
```
```
mvn test
```
## Actual behaviour
`target/surefire-reports/OuterTest.txt`:
```
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 -- in OuterTest
```
`target/surefire-reports/TEST-OuterTest.xml`:
```xml
```
## Expected behaviour
`OuterTest.txt` should report `Tests run: 3`, consistent with the XML.
## Impact
- CI tooling or reporting tools that consume `.txt` files per class see 0 tests where there are actually N — results appear to silently vanish.
- When `@BeforeEach` on the outer class throws, the XML shows errors but the TXT still shows 0, making errors invisible in the text report.
## Root cause (analysis from SUREFIRE-2194)
`RunListenerAdapter` does not distinguish the outer class from its nested classes — each starts a new test set. `TestSetRunListener` does not support nested test sets, so the nested class's test set supersedes the outer class's, which closes with 0 tests. The XML reporter aggregates correctly across the hierarchy; the TXT reporter does not.
## Suggested fix
Two approaches are viable:
**Option 1 — Emit a separate TXT report per nested class**, mirroring how the XML reporter already produces `TEST-OuterTest$GroupA.xml` entries. Each nested class gets its own `.txt` file with an accurate count. The outer class `.txt` would then correctly show 0 (it has no tests of its own) and tooling would find the results in the nested class files.
**Option 2 — Aggregate nested results into the top-level TXT report**, matching the current XML behaviour where all nested results roll up into `TEST-OuterTest.xml`. The outer class `.txt` would show the sum of all nested tests, giving a single accurate summary per top-level class.
Option 2 keeps the TXT and XML in sync by design. Option 1 is more granular but requires tooling to follow the `$`-separated nesting to find all results.
## Relation to prior issues
- **SUREFIRE-2194** (#2682, open since 3.1.2) — same root cause, different symptom: with tests at *both* outer and nested levels, results land in the wrong XML file. Our case (all tests nested, outer TXT shows 0 while XML is correct) is not covered there.
- **SUREFIRE-2298** (#2601, fixed in 3.5.4 via PR #828) — fixed a related XML regression for Cucumber/ArchUnit. The TXT inconsistency was not addressed by that fix.
Contributor guide
Research direction
Start by running mvn test in the linked minimal reproducer and inspecting the generated TXT and XML reports. Then trace RunListenerAdapter and TestSetRunListener, comparing their handling of nested test sets with the XML reporter. Done means the chosen TXT-report behavior accurately represents nested test results and is covered by regression tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100