HTTP caching should honor multiple IP addresses for the cache server, and load balance
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
When Bazel is set up to use an HTTP(/HTTPS) cache, it resolves the hostname passed with `--remote_cache` once, at startup, to a single IP address. It then uses that IP address consistently, for as long as the Bazel instance continues running.
Some caches are set up with multiple servers for load balancing, where a DNS query for the cache hostname returns multiple IP addresses. Ideally in this situation, since Bazel can end up opening 100 connections or more, it would load balance them across all the IP addresses advertised in DNS, rather than sending them all to just one.
(One scenario where this can be particularly helpful is when the clients are behind a misbehaving DNS cache: usually the DNS server will try to force load balancing by returning the IP addresses in a random order - but if a particular ordering gets stuck in the cache, then all clients will see the same one, and (with the current Bazel behavior) will all end up using the same IP address - with the result that one cache server melts, while the other(s) sit idle.)
### Which category does this issue belong to?
Remote Execution
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Set up two cache servers, with different IP addresses. Ensure that the DNS entry for the hostname returns both of the IP addresses. Specify a `--remote_cache` URL pointing to that hostname. Run a build that tries to retrieve lots of entries from the cache.
Observe that all cache requests go to just one of the servers, rather than being shared between them.
### Which operating system are you running Bazel on?
Linux (RHEL 8.9)
### What is the output of `bazel info release`?
release 8.0.0- (@non-git)
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
Using the command:
```
env SOURCE_DATE_EPOCH=1 EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh > compile.log
```
and the following dependencies:
* OpenJDK 12.0.3+9 from [here](https://github.com/adoptium/temurin21-binaries/releases/tag/jdk-21.0.3%2B9)
* Python 3.9.7
### What's the output of `git remote get-url origin; git rev-parse HEAD` ?
```text
https://github.com/bazelbuild/bazel
24cba660a7231786405ac40335c2e7e5bd4d6859
```
### If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
N/A
### Have you found anything relevant by searching the web?
_No response_
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
Start with the --remote_cache path and reproduce the issue using two cache servers behind a hostname whose DNS response contains both IP addresses. Trace how Bazel resolves the hostname and sends HTTP cache requests during a build. Done means requests are shared across the advertised addresses rather than consistently sent to one server.
Written by the indexing model from the issue text.
Assessment
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100