bazelbuild / bazelbuild/bazel

HTTP caching should honor multiple IP addresses for the cache server, and load balance

Open
#25,943 2 comments 1 reaction 0 assignees View on GitHub
P4 team-Remote-Exec type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

When Bazel is set up to use an HTTP(/HTTPS) cache, it resolves the hostname passed with `--remote_cache` once, at startup, to a single IP address. It then uses that IP address consistently, for as long as the Bazel instance continues running.

Some caches are set up with multiple servers for load balancing, where a DNS query for the cache hostname returns multiple IP addresses. Ideally in this situation, since Bazel can end up opening 100 connections or more, it would load balance them across all the IP addresses advertised in DNS, rather than sending them all to just one.

(One scenario where this can be particularly helpful is when the clients are behind a misbehaving DNS cache: usually the DNS server will try to force load balancing by returning the IP addresses in a random order - but if a particular ordering gets stuck in the cache, then all clients will see the same one, and (with the current Bazel behavior) will all end up using the same IP address - with the result that one cache server melts, while the other(s) sit idle.)

### Which category does this issue belong to?

Remote Execution

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Set up two cache servers, with different IP addresses. Ensure that the DNS entry for the hostname returns both of the IP addresses. Specify a `--remote_cache` URL pointing to that hostname. Run a build that tries to retrieve lots of entries from the cache.

Observe that all cache requests go to just one of the servers, rather than being shared between them.

### Which operating system are you running Bazel on?

Linux (RHEL 8.9)

### What is the output of `bazel info release`?

release 8.0.0- (@non-git)

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

Using the command:
```
env SOURCE_DATE_EPOCH=1 EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh > compile.log
```
and the following dependencies:
* OpenJDK 12.0.3+9 from [here](https://github.com/adoptium/temurin21-binaries/releases/tag/jdk-21.0.3%2B9)
* Python 3.9.7

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

```text
https://github.com/bazelbuild/bazel
24cba660a7231786405ac40335c2e7e5bd4d6859
```

### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

N/A

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the --remote_cache path and reproduce the issue using two cache servers behind a hostname whose DNS response contains both IP addresses. Trace how Bazel resolves the hostname and sends HTTP cache requests during a build. Done means requests are shared across the advertised addresses rather than consistently sent to one server.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.