bazelbuild / bazelbuild/bazel

Bazel fails the build with error code 1 on remote cache timeout

Open
#22,356 0 comments 0 reactions 0 assignees View on GitHub
help wanted P2 team-Remote-Exec type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

We have a rarely reproduced issue when Bazel fails to build the target because of remote cache timeout error. This is probably caused by network issues or high load of remote cache server. The error message in log is like following:

`ERROR: /Users/buildadmin/a/c/g_DCJ9SF26/r/*****/BUILD.bazel:73:24: Compiling *****/event.cpp failed: unable to finalize action: Download of '/cas/0535e6bae6fd101e71c26d71b14b7f1af7bd0abbb98189318dd1c8b9ddfcb4f9' timed out. Received 0 bytes.`

After that Bazel exits with error code **1**, which means "BUILD_FAILURE". This is unexpected.
We would expect that Bazel either falls back to local execution on remote cache errors or fails with error code like **32** ("REMOTE_ENVIRONMENTAL_ERROR"), **34** ("REMOTE_ERROR") or **39** ("REMOTE_CACHE_EVICTED"). In that case we would be able to handle the cache errors outside the Bazel and restart the build if needed.
With existing behavior, we are not able to identify the reason of failure and process it properly.

### Which category does this issue belong to?

Local Execution

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

_No response_

### Which operating system are you running Bazel on?

Linux, macOS

### What is the output of `bazel info release`?

release 7.0.2

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

_No response_

### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

Log files:
[linux.log](https://github.com/bazelbuild/bazel/files/15300895/linux.log)
[macos.log](https://github.com/bazelbuild/bazel/files/15300897/macos.log)

I guess that `BulkTransferException` should be handled separately from `IOException` in
https://github.com/bazelbuild/bazel/blob/bf2f762f815cc5fd0a4310c66f1633e415177521/src/main/java/com/google/devtools/build/lib/skyframe/SkyframeActionExecutor.java#L1263

Contributor guide

Open the contributing guide

Research direction

Start with src/main/java/com/google/devtools/build/lib/skyframe/SkyframeActionExecutor.java around line 1263 and review the timeout output in linux.log and macos.log. Investigate how BulkTransferException is handled during remote-cache downloads. Done should include a reproducible case and behavior that distinguishes the remote-cache failure from a generic BUILD_FAILURE.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.