bazelbuild / bazelbuild/bazel

Feature request: remote cache: Don't wait for uploads to finish

Open
#4,757 9 comments 10 reactions 0 assignees View on GitHub
not stale P2 team-Remote-Exec type: feature request
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the problem / feature request:

It would be nice if, when Bazel is uploading something to the remote cache, that process was nonblocking: The rest of the build proceeded while the upload was taking place, and when the build was done, the command prompt returned even though the upload was still happening in the background.

### Feature requests: what underlying problem are you trying to solve with this feature?

For people who do have slow or moderate-speed Internet connections, the upload part of the bazel cache can actually make builds dramatically slower.

For example, if I am at home, my upload speed is 11 megabits per second. If my build process creates, say, a 50MB file (such as the output of `@bazel_tools//tools/jdk:gen_platformclasspath`), the upload takes 36 seconds. That is 36 seconds that is "wasted" for me, since the upload is just for the cache.

I realize this is a very tall order, so it may simply be too hard:

* without some kind of limits in place, the local machine could quickly develop a huge backlog of data to upload
* to deal with that, it might make sense to have the option to "drop" some uploads (since it's just a cache)
* you would probably have to copy the local files to another location before uploading, and delete it after, to avoid having the file change during upload
* currently, the foreground bazel, not the daemon, does the uploading, so returning to the command prompt before finishing the upload would require changing that somehow

Probably a much more feasible solution than modifying Bazel itself is to have a proxy that sits between bazel and the actual remote cache. It could then handle the above problems. Bazel would think it had uploaded successfully; the (long-lived) proxy would upload to the actual remote cache, and would handle the above issues. But nonetheless, I wanted to offer this as a feature suggestion.

### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

`BUILD` file:

```py
genrule(
name = "main",
srcs = ["in.txt"],
outs = ["out.txt"],
cmd = "cp $(SRCS) $@",
)
```

Set up a remote cache. Then, at the command line: generate a 50MB dummy file, and then run bazel, which will end up copying a 50MB build artifact to the remote cache:

```bash
$ dd if=/dev/urandom of=in.txt count=50000 bs=1000
$ time bazel build --remote_http_cache=... --experimental_remote_spawn_cache :main
```

### What operating system are you running Bazel on?

macOS 10.13.3

### What's the output of `bazel info release`?

0.11.0

### Have you found anything relevant by searching the web?

GitHub issues, and google: "bazel upload wait"

Contributor guide

Open the contributing guide

Research direction

Reproduce the reported behavior with the provided BUILD genrule, a 50MB input from the dd command, and bazel build using --remote_http_cache and --experimental_remote_spawn_cache. Start by tracing the remote-cache upload path; done would mean the build and command prompt return without waiting for uploads, while limits, dropped uploads, and file lifetime remain safely handled.

Written by the indexing model from the issue text.

Assessment

Domain
build-system
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.