bazel-contrib / bazel-contrib/rules_jvm
Don't fail silently when running a junit4 test in junit5 test suite w/o vintage
- Dominant language
- Go
- Stars
- 54
- Forks
- 98
- Avg merge
- 6d 2h
- Merged PRs (30d)
- 7
Description
I am recently working on migrating a project from Maven to Bazel. As part of this, it has most modules which use junit4, but some which use junit5, and this ruleset really helps for the latter (thank you!).
However, we do have some modules which used both junit4 and junit5 classes (which was possible with maven-surefire-plugin). When migrating to Bazel, we use the `junit5_test_suite` rule, and we can either update all these classes to use JUnit 5, or use the vintage runner [which is all fine :)]
But, my main problem comes when using the `junit5_test_suite` but *forgetting* to also set up the vintage runner. Somewhat surprisingly, neither junit5 nor the bazel test runner fails in this case, and instead silently passes (with 0 test cases run). This seems like a potential concern, if developers assume their test is running but actually it wasn't but just passed anyway.
I did see that there is actually an open issue in junit5 to better report errors for invalid setups (https://github.com/junit-team/junit5/issues/1223 and https://github.com/junit-team/junit5/issues/242), but unfortunately there doesn't seem to be much movement there. I was curious if this was something detect-able via the the bazel test runner. Something along the lines of how bazel errors with "no test targets found, but testing was requested", except for the junit5 test runner instead.
Totally reasonable if not possible, but was interested to know.
Thanks for your time!
Contributor guide
Research direction
Start by tracing the junit5_test_suite runner and reproduce a suite containing JUnit 4 tests without the Vintage runner. Determine how the Bazel test result is produced when zero tests run; done means the setup emits a clear failure or diagnostic instead of passing silently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100