Bazel remote cache is not a clear win
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 72
Description
### Description of the ~problem~ / feature request:
Applying a Bazel HTTP remote cache can make the build _slower_, depending on the project and artefacts being built.
I tried a few projects:
* **Cartographer** ~2.8x _faster_ with cache
* **Abseil** ~1.5x _faster_ with cache
* **cppitertools** ~2x _slower_ with cache
* **OpenTracing** ~2x _slower_ with cache
For the cache server, I tried both [bazel-remote](https://github.com/buchgr/bazel-remote) and my own Node.js server that I cobbled together. Both yielded similar results.
The cache was hosted on a reasonable Digital Ocean box in the same city:
```bash
$ less /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
stepping : 4
microcode : 0x1
cpu MHz : 2294.608
...
$ free -m
total used free shared buff/cache available
Mem: 3944 119 153 0 3672 3544
Swap: 0 0 0
```
Internet connection speed for the client was around **10mbps**, latency **~20ms**. Not the fastest, but Bazel should be able to adapt to this.
The Bazel client was running on a fairly high-end laptop:
```bash
$ less /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 142
model name : Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
stepping : 10
microcode : 0x9a
cpu MHz : 700.060
cache size : 8192 KB
...
$ free -m
total used free shared buff/cache available
Mem: 15806 4246 7541 755 4018 10582
Swap: 32767 0 32767
```
**Suggestion**
Perhaps the HTTP cache should record the time it took to build an artefact (according to the client). This would give Bazel enough information to decide if it is better to build or fetch.
Relevant variables:
* Connection speed
* Size of artefact
* Estimated time to build artefact
* Current build workload
Currently the server receives minimal metadata from Bazel.
### What operating system are you running Bazel on?
Ubuntu 18.10
### What's the output of `bazel info release`?
```bash
release 0.23.1
```
### Have you found anything relevant by searching the web?
Related discussion: https://github.com/bazelbuild/bazel/issues/6091
Contributor guide
Research direction
No Bazel files, tests, or entry points are named. Start with the related discussion in issue #6091 and review how the remote cache handles artifact requests and client metadata. Done means establishing whether build-versus-fetch decisions can use measured build time, artifact size, connection speed, and current workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nodejs
- Domain
- build-system, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100