bazelbuild / bazelbuild/bazel

BEP could not fall back to local with "--build_event_json_file" flag

Open
#25,542 3 comments 0 reactions 0 assignees View on GitHub
team-Core type: bug untriaged
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

Seems `--build_event_json_file` flag could not play well with `--remote_local_fallback`.

When RBE service is not available, build and test can fall back to local with `--remote_local_fallback`. However, BEP upload still tried to talk to remote executor and fails with exit code 38
```
WARNING: Uploading BEP referenced local file /private/var/tmp/_bazel_tawang/712513e1d334efe00c38c876880da33f/command-8570ef85-b571-4cd2-8317-690614d537ed.profile.gz hash: "dda84ff333d044c2d4c7ebca3751e12dbdb348f71ac4cfe4e6caa5da8c5ae0b8"
size_bytes: 20030
: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/[0:0:0:0:0:0:0:1]:8980
```

The cause is that `ConnectivityStatusProvider` was not implemented and always returned `OK` ([code](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/network/NoOpConnectivityModule.java#L23)), so `buildEventUploadStrategy` ([code](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventservice/BuildEventServiceModule.java#L364)) could not automatically fall back to local when RBE service is down.

### Which category does this issue belong to?

Remote Execution

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

With the attached sample project below

`bazelisk clean && bazelisk build //... --config remote-common --build_event_json_file=foo` failed with the error below and exit code 38.
```
WARNING: Uploading BEP referenced local file /private/var/tmp/_bazel_tawang/712513e1d334efe00c38c876880da33f/command-8570ef85-b571-4cd2-8317-690614d537ed.profile.gz hash: "dda84ff333d044c2d4c7ebca3751e12dbdb348f71ac4cfe4e6caa5da8c5ae0b8"
size_bytes: 20030
: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/[0:0:0:0:0:0:0:1]:8980

echo $?
38
```

With the extra flag `--experimental_build_event_upload_strategy=local` it could pass
```
 bazelisk clean && bazelisk build //... --config remote-common --build_event_json_file=foo --experimental_build_event_upload_strategy=local
INFO: Starting clean (this may take a while). Use --async if the clean takes more than several minutes.
INFO: Invocation ID: a4e4d032-2e7a-41f4-9e57-0f522c11d88a
INFO: Analyzed target //:main (68 packages loaded, 475 targets configured).
INFO: Found 1 target...
Target //:main up-to-date:
bazel-bin/main
INFO: Elapsed time: 3.996s, Critical Path: 3.40s
INFO: 9 processes: 7 internal, 2 local.
INFO: Build completed successfully, 9 total actions
INFO: Build Event Protocol files produced successfully.
```

---

Sample project is as below

`.bazelrc`
```
build:remote-common --jobs=200
build:remote-common --define=EXECUTOR=remote
build:remote-common --spawn_strategy=remote,sandboxed,local
# Disable disk cache, Bazel client should talk to the cache service directly
build:remote-common --disk_cache=
build:remote-common --incompatible_strict_action_env=true
# Avoid hanging if the remote side closes the connection without a TCP RST. Only
# sends a keepalive if we don't receive any messages (server updates every 60s).
build:remote-common --grpc_keepalive_time=10s
build:remote-common --remote_timeout=60
build:remote-common --remote_executor=grpc://localhost:8980
build:remote-common --remote_local_fallback
```

`BUILD.bazel`
```
cc_binary(
name = "main",
srcs = ["main.cc"],
)
```

`main.cc`
```
#include

int main( int argc, char *argv[] )
{
std::cout << "Hello, World!" << std::endl;
}
```

### Which operating system are you running Bazel on?

macOS 15.3.1

### What is the output of `bazel info release`?

release 8.1.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text

```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with src/main/java/com/google/devtools/build/lib/network/NoOpConnectivityModule.java and the buildEventUploadStrategy logic in src/main/java/com/google/devtools/build/lib/buildeventservice/BuildEventServiceModule.java. Reproduce with the provided remote-common configuration, an unavailable localhost:8980 executor, --remote_local_fallback, and --build_event_json_file=foo. Done means the build and BEP upload fall back locally and complete successfully without exit code 38.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
build-system, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.