bazel-contrib / bazel-contrib/rules_jvm

Don't fail silently when running a junit4 test in junit5 test suite w/o vintage

Open
#277 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
54
Forks
98
Avg merge
6d 2h
Merged PRs (30d)
7

Description

I am recently working on migrating a project from Maven to Bazel. As part of this, it has most modules which use junit4, but some which use junit5, and this ruleset really helps for the latter (thank you!).

However, we do have some modules which used both junit4 and junit5 classes (which was possible with maven-surefire-plugin). When migrating to Bazel, we use the `junit5_test_suite` rule, and we can either update all these classes to use JUnit 5, or use the vintage runner [which is all fine :)]

But, my main problem comes when using the `junit5_test_suite` but *forgetting* to also set up the vintage runner. Somewhat surprisingly, neither junit5 nor the bazel test runner fails in this case, and instead silently passes (with 0 test cases run). This seems like a potential concern, if developers assume their test is running but actually it wasn't but just passed anyway.

I did see that there is actually an open issue in junit5 to better report errors for invalid setups (https://github.com/junit-team/junit5/issues/1223 and https://github.com/junit-team/junit5/issues/242), but unfortunately there doesn't seem to be much movement there. I was curious if this was something detect-able via the the bazel test runner. Something along the lines of how bazel errors with "no test targets found, but testing was requested", except for the junit5 test runner instead.

Totally reasonable if not possible, but was interested to know.
Thanks for your time!

Contributor guide

Open the contributing guide

Research direction

Start by tracing the junit5_test_suite runner and reproduce a suite containing JUnit 4 tests without the Vintage runner. Determine how the Bazel test result is produced when zero tests run; done means the setup emits a clear failure or diagnostic instead of passing silently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.