bazelbuild / bazelbuild/rules_java

Improve performance when running a large number of JUnit tests

オープン
#172 コメント 1 件 リアクション 1 件 担当者 0 名 GitHub で見る
P2
主要言語
Starlark
スター
103
フォーク
102
PR マージ指標
30日以内にマージされた PR はありません

説明

Title is a bit vague here, apologies for that.

Out-of-the-box, it is only possible to provide a single entry point to `java_test`, e.g.

```
java_test(
name = "com.example.MyLovelyUnitTest",
test_class = "com.example.MyLovelyUnitTest",
srcs = [ ... ],
# etc etc
)
```

If one wishes to make a target from multiple classes, there are currently two well publicised workarounds:

1. Use a macro wrapper;
2. Use a `@Suite` or similar.

Unfortunately, each of these comes with a negative side effect wrt. performance:

1. Using a macro wrapper to turn the glob of source files into distinct `java_test` targets [is documented](https://github.com/bazelbuild/bazel/issues/7595) to have a significant performance impact due to having to create and tear-down workers. To give a rough idea of the impact of this, here are some numbers taken from a Bazel project with ~5000 unit tests using a `@Suite`:
```
$ bazel clean
$ time bazel test //... --build_tests_only --test_lang_filters=java --test_size_filters=small
...
Executed 498 out of 498 tests: 498 tests pass.
...
bazel test //... --build_tests_only --test_lang_filters=java 2.58s user 1.55s system 0% cpu 16:41.63 total
```

... vs using a macro wrapper + aggregate `test_suite`:
```
$ bazel clean
$ time bazel test //... --build_tests_only --test_lang_filters=java --test_size_filters=small
...
Executed 4691 out of 4691 tests: 4691 tests pass.
...
./bb bazel bz test //... --build_tests_only --test_lang_filters=java 2.72s user 1.69s system 0% cpu 26:47.95 total
```

Both of these runs were operating over the same set of tests, but using a separate `java_test` for each individual class causes the build time to increase by ~60%.

2. Generating a `@Suite` (either at compile time or dynamically via something like `AllTests`) clashes with `--flaky_test_attempts` as, if any test case fails, the entire suite is detected as having failed and so _all_ tests are run again. This can be somewhat mitigated by sharding but there's a cap of 50 on the number of shards.

Option 1 is "okay" in cases where there are few, long-running tests; option 2 is "okay" for a lot of fast running tests. It would be nice to have a "one-size-fits-all" solution.

There has been an [open issue in the main Bazel repository](https://github.com/bazelbuild/rules_java/issues/330) for a few years now that has a bit of overlap, but that one seems a bit more focused around convenience rather than performance. As the Java rules are being broken out, I thought it might make sense to move it over here for an updated discussion.

When I came across the original issue, I did a very [quick-and-hacky PoC](https://github.com/swarren12/bazel/commit/8e090ee44dc3d97cee0e860c03bf2ced3491d2f2) of how the built-in Bazel test runner could be updated to support multiple classes; however, on revisiting this I'm not sure if that would _actually_ solve the performance problem on its own, as it looks as though the flaky test attempts is handled outside the test runner process, and so I'm guessing this solution would end up just working in the same way as a `@Suite`.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start by tracing the java_test entry point and the Bazel test runner behavior described in the issue, especially how macro-generated targets, @Suite aggregation, sharding, and --flaky_test_attempts interact. Compare both reported test runs and the linked proof of concept; done should address large test collections without the reported performance increase or rerunning every test after a failure.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
java
領域
build-system, performance, testing-qa
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
説明が足りない
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。