bazel-contrib / bazel-contrib/toolchains_llvm

Using sysroot rule with remote or disk cache causes mass action cache invalidation on server restart

Open
#740 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Starlark
Stars
371
Forks
283
Avg merge
1d 55m
Merged PRs (30d)
25

Description

After adding some heavy third-party dependencies to our repo, I noticed a lot of them needed to be rebuilt or redownloaded from the cache when the Bazel server restarted. I narrowed the issue down to the toolchains_llvm sysroot rule.

````
$ bazel build ...
...
$ bazel shutdown
$ bazel build ...
Starting local Bazel server (8.5.1) and connecting to it...
INFO: Invocation ID: 4a671a9b-643e-40ac-bafa-feabeb678305
INFO: Analyzed 4 targets (114 packages loaded, 6142 targets configured).
INFO: Found 4 targets...
INFO: Elapsed time: 5.657s, Critical Path: 1.67s
INFO: 50 processes: 287 action cache hit, 49 disk cache hit, 1 internal.
INFO: Build completed successfully, 50 total actions
````
^ bazel reaches out to a on disk or remote cache instead of the action cache, even though nothing changed.

I've recreated the issue here:

https://github.com/jemeador/bug_repro_toolchains_llvm_remote_cache

Potential bug fix here, but I don't know enough about the context of why `srcs = ["."]` was used instead of `glob` to open a PR. After making this change, `bazel shutdown` no longer prevents 100% `action_cache` hits.

````
$ bazel shutdown
$ bazel build ...
Starting local Bazel server (8.5.1) and connecting to it...
INFO: Invocation ID: 73a78277-c639-460c-9a80-f7474ce29feb
INFO: Analyzed 4 targets (115 packages loaded, 24360 targets configured).
INFO: Found 4 targets...
INFO: Elapsed time: 4.137s, Critical Path: 0.42s
INFO: 1 process: 336 action cache hit, 1 internal.
INFO: Build completed successfully, 1 total action
````

https://github.com/jemeador/bug_repro_toolchains_llvm_remote_cache/pull/1

Contributor guide

Open the contributing guide

Research direction

Start with the toolchains_llvm sysroot rule and the `srcs = ["."]` entry mentioned in the report, then reproduce the behavior with the linked repository using `bazel build` and `bazel shutdown`. Compare action-cache results after restarting with remote or disk caching enabled. Done means the reproduction no longer causes the reported cache invalidation after a server restart.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, compilers
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.