bazelbuild / bazelbuild/bazel

Integration tests are not cached in presubmit on trivial change

Open
#16,518 2 comments 0 reactions 0 assignees View on GitHub
area-EngProd P2 team-OSS type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

There is only a change in a [java test file](https://bazel-review.googlesource.com/c/bazel/+/205878/4/src/test/java/com/google/devtools/build/lib/runtime/UiEventHandlerStdOutAndStdErrTest.java) between two Bazel presubmits runs:
- https://buildkite.com/bazel/google-bazel-presubmit/builds/60390
- https://buildkite.com/bazel/google-bazel-presubmit/builds/60386

Ideally, only that particular changed java test should rerun and all other tests should be cached. But for various reasons, many integration tests were also triggered on the trivial change (aggregated from shared test jobs):
```
//src/test/shell/bazel:bazel_bootstrap_distfile_tar_test PASSED in 271.3s
//src/test/shell/bazel:bazel_java_test_jdk11_toolchain_head PASSED in 126.1s
//src/test/shell/bazel:bazel_java_tools_test PASSED in 88.8s
//src/test/shell/bazel/android:aar_integration_test PASSED in 247.7s
//src/test/shell/bazel/android:aidl_integration_test_with_head_android_tools PASSED in 50.0s
//src/test/shell/bazel/android:desugarer_integration_test_with_platforms PASSED in 208.0s
//src/test/shell/bazel/android:resource_processing_integration_test PASSED in 90.3s
//src/tools/singlejar:output_jar_simple_test PASSED in 4.4s
//src/test/shell/bazel/android:android_sdk_integration_test_with_head_android_tools PASSED in 56.5s
//src/test/shell/bazel/android:android_instrumentation_test_integration_test_with_platforms PASSED in 142.3s
//src/test/shell/bazel/android:android_ndk_integration_test PASSED in 230.0s
//src/test/shell/bazel:bazel_bootstrap_distfile_test PASSED in 315.1s
//src/test/shell/bazel:bazel_coverage_java_jdk17_toolchain_head_test PASSED in 135.7s
//src/test/shell/bazel:bazel_java17_test PASSED in 36.2s
//src/test/shell/bazel:bazel_java_test_jdk17_toolchain_head PASSED in 257.8s
//src/test/shell/bazel/android:aapt_integration_test PASSED in 44.2s
//src/test/shell/bazel/android:aar_integration_test_with_head_android_tools PASSED in 235.5s
//src/test/shell/bazel/android:aidl_integration_test_with_platforms PASSED in 41.2s
//src/test/shell/bazel/android:android_integration_test PASSED in 467.5s
//src/test/shell/bazel/android:proguard_integration_test PASSED in 120.4s
//src/test/shell/bazel/android:resource_processing_integration_test_with_head_android_tools PASSED in 243.5s
//src/test/shell/bazel/android:android_sdk_integration_test_with_platforms PASSED in 177.5s
//src/test/shell/bazel/android:android_ndk_integration_test_with_head_android_tools PASSED in 223.1s
//src/test/shell/bazel:bazel_determinism_test PASSED in 395.5s
//src/test/shell/bazel:jdeps_test PASSED in 145.5s
//src/test/shell/bazel/android:aapt_integration_test_with_head_android_tools PASSED in 151.8s
//src/test/shell/bazel/android:aar_integration_test_with_platforms PASSED in 285.5s
//src/test/shell/bazel/android:android_integration_test_with_head_android_tools PASSED in 458.4s
//src/test/shell/bazel/android:desugarer_integration_test PASSED in 215.8s
//src/test/shell/bazel/android:proguard_integration_test_with_head_android_tools PASSED in 166.4s
//src/test/shell/bazel/android:resource_processing_integration_test_with_platforms PASSED in 266.9s
//src/test/shell/bazel/android:android_instrumentation_test_integration_test PASSED in 60.2s
//src/test/shell/bazel/android:android_ndk_integration_test_with_platforms PASSED in 214.4s
//src/test/java/com/google/devtools/build/lib:runtime-tests PASSED in 14.6s
//src/test/shell/bazel:bazel_coverage_java_jdk11_toolchain_head_test PASSED in 90.3s
//src/test/shell/bazel/android:aapt_integration_test_with_platforms PASSED in 89.3s
//src/test/shell/bazel/android:aidl_integration_test PASSED in 96.3s
//src/test/shell/bazel/android:android_integration_test_with_platforms PASSED in 307.8s
//src/test/shell/bazel/android:desugarer_integration_test_with_head_android_tools PASSED in 120.6s
//src/test/shell/bazel/android:proguard_integration_test_with_platforms PASSED in 93.7s
//src/test/shell/bazel/android:android_sdk_integration_test PASSED in 96.9s
//src/test/shell/bazel/android:android_instrumentation_test_integration_test_with_head_android_tools PASSED in 87.6s
```

We should investigate and make sure remote cache should work whenever possible.

For android tests, related issues are:
- https://github.com/bazelbuild/bazel/issues/8235
- https://github.com/bazelbuild/bazel/issues/4663

Contributor guide

Open the contributing guide

Research direction

Compare Buildkite presubmit builds 60390 and 60386, starting with the changed src/test/java/com/google/devtools/build/lib/runtime/UiEventHandlerStdOutAndStdErrTest.java and the listed integration-test targets. Trace the presubmit and remote-cache configuration to find why unrelated shared test jobs rerun. Done means the changed test runs while unaffected integration tests use the remote cache whenever possible.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.