bazelbuild / bazelbuild/bazel

Maintain multiple analyses in analysis cache

Open
#14,179 20 comments 18 reactions 0 assignees View on GitHub
P3 team-Configurability type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 18h
Merged PRs (30d)
75

Description

### Description of the problem / feature request:

Bazel seems to maintain only one analysis in the analysis cache. Whenever a relevant flag is changed, we purge the cache in its entirety. This adds to build time even with a 100% cache hit rate for the build artifacts (see repro below). Workarounds I've seen so far suggest using a different output_base for each set of flags, which is pretty cumbersome and uses a ton of cache space. Envoy's gotten around by sharing the same [set of flags](https://gitlab.cncf.ci/envoyproxy/envoy/commit/49277cebb22ad584fff375de2c31abfd381aa30d) for both build and test, but that's not always possible and is a stop gap for bazel preserving earlier analyses in the analysis cache for possible future re-use.

### Feature requests: what underlying problem are you trying to solve with this feature?

It's a common workflow to switch between building and testing a bazel project. For us that entails using different `--define` and `--test_env` flags, which discards the analysis cache entirely. If bazel's maintaining only one copy of the analysis cache (from the last bazel invocation), we end up doing a lot of extra work to re-analyze the build despite 100% of the build artifacts being present in the remote cache. Maintaining multiple analyses in the analysis cache, each tagged with whatever compiler options they're safe to use with, would help us cut the iteration time down substantially. This also holds true for CI, where even if all the necessary artifacts are present, thrashing the analysis cache results in us doing a lot of unnecessary work.

### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
With https://github.com/cockroachdb/cockroach:

```
$ bazel build //pkg/cmd/cockroach-short:cockroach-short
INFO: Build options --define and --test_env have changed, discarding analysis cache.
INFO: Analyzed target //pkg/cmd/cockroach-short:cockroach-short (0 packages loaded, 18793 targets configured).
INFO: Found 1 target...
Target //pkg/cmd/cockroach-short:cockroach-short up-to-date:
_bazel/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short
INFO: Elapsed time: 25.610s, Critical Path: 20.89s
INFO: 1995 processes: 1994 disk cache hit, 1 internal.
INFO: Build completed successfully, 1995 total actions
Successfully built binary for target //pkg/cmd/cockroach-short:cockroach-short at cockroach-short

$ bazel test //pkg/spanconfig/spanconfigkvsubscriber:spanconfigkvsubscriber_test --test_filter=TestDataDriven/basic --test_output errors
INFO: Build options --define and --test_env have changed, discarding analysis cache.
INFO: Analyzed target //pkg/spanconfig/spanconfigkvsubscriber:spanconfigkvsubscriber_test (0 packages loaded, 17487 targets configured).
INFO: Found 1 test target...
Target //pkg/spanconfig/spanconfigkvsubscriber:spanconfigkvsubscriber_test up-to-date:
_bazel/bin/pkg/spanconfig/spanconfigkvsubscriber/spanconfigkvsubscriber_test_/spanconfigkvsubscriber_test
INFO: Elapsed time: 26.116s, Critical Path: 24.73s
INFO: 1995 processes: 1994 disk cache hit, 1 internal.
INFO: Build completed successfully, 1995 total actions
//pkg/spanconfig/spanconfigkvsubscriber:spanconfigkvsubscriber_test (cached) PASSED in 1.8s

Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1995 total actions

$ bazel build //pkg/cmd/cockroach-short:cockroach-short
WARNING: Ignoring JAVA_HOME, because it must point to a JDK, not a JRE.
INFO: Invocation ID: 7822bf1c-b690-4138-bdf9-45a4b1eed074
INFO: Build options --define and --test_env have changed, discarding analysis cache.
INFO: Analyzed target //pkg/cmd/cockroach-short:cockroach-short (0 packages loaded, 18793 targets configured).
INFO: Found 1 target...
Target //pkg/cmd/cockroach-short:cockroach-short up-to-date:
_bazel/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short
INFO: Elapsed time: 25.610s, Critical Path: 20.89s
INFO: 1995 processes: 1994 disk cache hit, 1 internal.
INFO: Build completed successfully, 1995 total actions
Successfully built binary for target //pkg/cmd/cockroach-short:cockroach-short at cockroach-short
```

Observe that when switching between `bazel build` and `bazel test`, despite have a 100% disk cache hit, it takes ~25s for the execution to complete as a result of not having an earlier run's analysis cache data to consult. Compare it to when it is available:

```
$ bazel build //pkg/cmd/cockroach-short:cockroach-short --profile=build.gz
INFO: Analyzed target //pkg/cmd/cockroach-short:cockroach-short (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //pkg/cmd/cockroach-short:cockroach-short up-to-date:
_bazel/bin/pkg/cmd/cockroach-short/cockroach-short_/cockroach-short
INFO: Elapsed time: 1.259s, Critical Path: 0.83s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
Successfully built binary for target //pkg/cmd/cockroach-short:cockroach-short at cockroach-short
```

### What operating system are you running Bazel on?

MacOS Big Sur.

### What's the output of `bazel info release`?

release 6.0.0-pre.20211019.1, though the same problem exists with 4.0.0 onwards.

### Have you found anything relevant by searching the web?

Some other threads on the mailing list: https://groups.google.com/g/bazel-discuss/c/EdBvMEPrH5A and https://groups.google.com/g/bazel-discuss/c/vgn9uyyIrIM/m/4trNkRS0AQAJ.
Github: #11194, https://github.com/bazelbuild/bazel/issues/12113#issuecomment-952602517.
Stack Overflow: https://stackoverflow.com/questions/53012722/why-does-bazel-do-a-full-rebuild-whenever-switching-between-intellij-and-command

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the alternating bazel build and bazel test commands with different --define and --test_env values, then compare them with the repeated build that reports 0 packages loaded and 0 targets configured. Trace how Bazel handles the analysis cache when build options change. Done means multiple compatible analyses can be retained and reused without discarding the entire cache.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.