BEP could not fall back to local with "--build_event_json_file" flag
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
Seems `--build_event_json_file` flag could not play well with `--remote_local_fallback`.
When RBE service is not available, build and test can fall back to local with `--remote_local_fallback`. However, BEP upload still tried to talk to remote executor and fails with exit code 38
```
WARNING: Uploading BEP referenced local file /private/var/tmp/_bazel_tawang/712513e1d334efe00c38c876880da33f/command-8570ef85-b571-4cd2-8317-690614d537ed.profile.gz hash: "dda84ff333d044c2d4c7ebca3751e12dbdb348f71ac4cfe4e6caa5da8c5ae0b8"
size_bytes: 20030
: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/[0:0:0:0:0:0:0:1]:8980
```
The cause is that `ConnectivityStatusProvider` was not implemented and always returned `OK` ([code](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/network/NoOpConnectivityModule.java#L23)), so `buildEventUploadStrategy` ([code](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/buildeventservice/BuildEventServiceModule.java#L364)) could not automatically fall back to local when RBE service is down.
### Which category does this issue belong to?
Remote Execution
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
With the attached sample project below
`bazelisk clean && bazelisk build //... --config remote-common --build_event_json_file=foo` failed with the error below and exit code 38.
```
WARNING: Uploading BEP referenced local file /private/var/tmp/_bazel_tawang/712513e1d334efe00c38c876880da33f/command-8570ef85-b571-4cd2-8317-690614d537ed.profile.gz hash: "dda84ff333d044c2d4c7ebca3751e12dbdb348f71ac4cfe4e6caa5da8c5ae0b8"
size_bytes: 20030
: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/[0:0:0:0:0:0:0:1]:8980
echo $?
38
```
With the extra flag `--experimental_build_event_upload_strategy=local` it could pass
```
bazelisk clean && bazelisk build //... --config remote-common --build_event_json_file=foo --experimental_build_event_upload_strategy=local
INFO: Starting clean (this may take a while). Use --async if the clean takes more than several minutes.
INFO: Invocation ID: a4e4d032-2e7a-41f4-9e57-0f522c11d88a
INFO: Analyzed target //:main (68 packages loaded, 475 targets configured).
INFO: Found 1 target...
Target //:main up-to-date:
bazel-bin/main
INFO: Elapsed time: 3.996s, Critical Path: 3.40s
INFO: 9 processes: 7 internal, 2 local.
INFO: Build completed successfully, 9 total actions
INFO: Build Event Protocol files produced successfully.
```
---
Sample project is as below
`.bazelrc`
```
build:remote-common --jobs=200
build:remote-common --define=EXECUTOR=remote
build:remote-common --spawn_strategy=remote,sandboxed,local
# Disable disk cache, Bazel client should talk to the cache service directly
build:remote-common --disk_cache=
build:remote-common --incompatible_strict_action_env=true
# Avoid hanging if the remote side closes the connection without a TCP RST. Only
# sends a keepalive if we don't receive any messages (server updates every 60s).
build:remote-common --grpc_keepalive_time=10s
build:remote-common --remote_timeout=60
build:remote-common --remote_executor=grpc://localhost:8980
build:remote-common --remote_local_fallback
```
`BUILD.bazel`
```
cc_binary(
name = "main",
srcs = ["main.cc"],
)
```
`main.cc`
```
#include
int main( int argc, char *argv[] )
{
std::cout << "Hello, World!" << std::endl;
}
```
### Which operating system are you running Bazel on?
macOS 15.3.1
### What is the output of `bazel info release`?
release 8.1.0
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
```
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
_No response_
### Have you found anything relevant by searching the web?
_No response_
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
Start with src/main/java/com/google/devtools/build/lib/network/NoOpConnectivityModule.java and the buildEventUploadStrategy logic in src/main/java/com/google/devtools/build/lib/buildeventservice/BuildEventServiceModule.java. Reproduce with the provided remote-common configuration, an unavailable localhost:8980 executor, --remote_local_fallback, and --build_event_json_file=foo. Done means the build and BEP upload fall back locally and complete successfully without exit code 38.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- build-system, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100