bazelbuild / bazelbuild/bazel

BWOB build not robust to remote cache errors

Open
#18,393 6 comments 0 reactions 0 assignees View on GitHub
P2 team-Remote-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

When using `--remote_download_minimal`, our build hard-fails in some cases when our S3-based remote cache errors. However, builds without this flag seem able to recover from the same errors just fine (perhaps with local fallback or retries?).

My expectation is that builds with the flag be equally resilient to remote cache errors as those without.

NOTE: I am using remote_local_fallback.

```
ERROR: libs/comps/BUILD:15:13: Copying files to directory libs/comps/comps [for tool] failed: Exec failed due to IOException: 5 errors during bulk transfer:
java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
```

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

_No response_

### Which operating system are you running Bazel on?

Linux

### What is the output of `bazel info release`?

6.2.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reproducing a build with --remote_download_minimal and remote_local_fallback against the S3-based remote cache, then inspect the ClosedChannelException bulk-transfer failure. Done means remote cache errors recover through local fallback or retries as reliably as builds without --remote_download_minimal.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
build-system, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.