bazelbuild / bazelbuild/bazel

RBE "Failed to query remote execution capabilities" after proxy disconnect

Open
#10,204 13 comments 4 reactions 0 assignees View on GitHub
not stale P3 team-Remote-Exec type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the problem / feature request:

Currently if you're using Google's RBE service as a remote cache, you can end up in a state where you have to run `bazel shutdown` to recover your connection to the service.

### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

1. `bazel shutdown`
2. Launch [Charles Proxy](https://www.charlesproxy.com/) on macOS and enable the local macOS proxy
3. Run `bazel build --remote_cache=remotebuildexecution.googleapis.com --remote_instance_name=whatever //...
4. Quit Charles Proxy
5. Run your build again

At this point you are in a broken state where you only ever get this error:

```
ERROR: Failed to query remote execution capabilities: UNAVAILABLE: io exception
```

Until your run `bazel shutdown` again

### What operating system are you running Bazel on?

macOS

### What's the output of `bazel info release`?

release 1.1.0

### Have you found anything relevant by searching the web?

I've attempted to use `no_proxy` to avoid this, but it seems that that doesn't apply for grpc connections

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the bazel build sequence with Charles Proxy, quitting the proxy, and confirming that a second build fails until bazel shutdown runs. Trace the remote-cache/RBE gRPC connection handling from the bazel build entry point. Done means the build can recover after the proxy disconnect without requiring bazel shutdown, with coverage for the reported sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, grpc, java, macos
Domain
build-system, cloud, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.