bazelbuild / bazelbuild/bazel

Enabling --experimental_remote_cache_eviction_retries causes the same invocation to be used in certain cases

Open
#18,694 4 comments 0 reactions 0 assignees View on GitHub
P1 team-Remote-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 18h
Merged PRs (30d)
75

Description

### Description of the bug:

Hello! We have used `--experimental_remote_cache_eviction_retries=1` with `--remote_download_toplevel` during the last few weeks, and we noticed from our data that in some cases, when Bazel exits with exit code 39, the following invocation to recover will have the same invocation id. We see this because the first invocation is pushed to our BES as failing. Then in the CI logs we can see that the second invocation started with the same exact invocation id but passed, which make the final exit code of Bazel be successful. In our BES we can only see one failing build, because the second invocation was never sent to the BES backend due to having the same invocation id.

The logs that I see look like the following:
```
ERROR: /Users/path/BUILD.bazel:7:26: Linking libMyLib.a failed: Failed to fetch blobs because they do not exist remotely.: Missing digest: ccfb1a6ac214dab81868f41d8116d3928e25145edfbd777df7225bc70a2ad715/22056
....
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
INFO: Streaming build results to: https://my.remote-cache.io/invocation/128b1231-7b7c-4f9c-af9a-034ef759788d
Found remote cache eviction error, retrying the build...
INFO: Invocation ID: 128b1231-7b7c-4f9c-af9a-034ef759788d
INFO: Streaming build results to: https://my.remote-cache.io/invocation/128b1231-7b7c-4f9c-af9a-034ef759788d
...
INFO: Build completed successfully, 7047 total actions
```

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

We are not customizing `--invocation_id` ourselves, so I believe this to be a bug. In other invocations I can see that the `invocation_id` of the second build is different, so I don't think it's 100% reproducible. Let me know if I can help by providing more logs.

### Which operating system are you running Bazel on?

macOS

### What is the output of `bazel info release`?

6.2.0 @ 286306e8358542ce272f7442075bf157a2a62ec7

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

_No response_

### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the behavior with --experimental_remote_cache_eviction_retries=1 and --remote_download_toplevel on Bazel 6.2.0, then compare the invocation IDs in the retry logs and BES output. The issue is resolved when a recovered invocation receives the expected distinct ID and both build results are visible to the BES backend.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.