FR: disable re-downloading files to check hash is still the same
- Dominant language
- Rust
- Stars
- 4.4k
- Forks
- 394
- PR merge metrics
- No merged PRs in 30d
Description
Given an `http_archive`/`http_file`/`remote_file` rule, for example:
```starlark
# BUCK
remote_file(
name = "zip",
url = "https://static.crates.io/crates/windows-sys/0.59.0/download",
sha256 = "1e38bc4d79ed67fd075bcc251a1c39b32a1776bbe92e5bef1f0bf1f8c531853b",
)
genrule(
name = "genrule",
srcs = [":zip"],
out = "plain.txt",
cmd = "echo downloaded $SRCS > $OUT",
)
```
I have RE enabled, deferred materialization, sqlite materializer state, hash_all_commands = true, defer_write_actions = true. Buck always re-downloads the zip file after a clean. The genrule runs remotely, no local actions are running that need access to the file. And yet there it is in buck-out, 2.3 MB, every time. As I understand it, this is a feature to help buck discover ASAP when links have gone stale or have had their hash changed.
```sh
$ buck2
$ buck2 build :genrule
...
Downloaded: 2.3MB
$ buck2 clean
$ buck2 build :genrule
...
Downloaded: 2.3MB
```
### Mix that with huge generated BUCK files from reindeer
As I described in https://github.com/facebookincubator/reindeer/pull/46, buck having no kind of rate limiting or file descriptor limiting means every time you start a fresh buck daemon, it may involve thousands of concurrent requests to `static.crates.io` or a similar package CDN. That means CDNs hit you with their rate limiters, and buck often errors out with too many file descriptors when trying to save these downloaded files. This basically makes the un-vendored `http_archive` strategy unusable beyond a hundred or so crates, and that's what led to that reindeer PR.
I had been hoping that RE's cache would allow me to go back to http_archive. But it doesn't stop buck from pummeling thousands of requests at once on every daemon restart + once for every configuration.
CC @Ralith.
### What to do?
Buck appears to have the file cached in RE, I can't tell where it's downloading it from, but it is able to use a dependent target's action cache entry.
```
$ buck2 log what-ran
Showing commands from: buck2 build :genrule
build root//:genrule (cfg:#2ca3db971bb25d5c) (genrule) cache c69eeb7dda9a55d3c579516760611d1645af8857e07acc12d8e030dfaa59a979:142
```
So I think buck is downloading again from the source URL rather than materializing from cache.
I don't much care about discovering ASAP that a link has gone stale / gone offline. I'd rather find out later and have builds just keep working for a while, if as a tradeoff buck can be much faster and not hit the network after a clean. People are using http_archive for CDN urls like `https://static.crates.io/crates/mycrate/v0.2.2/download`, not `https://blog.soandso.com/my-latest-post`.
So, I'd like to be able to turn this off -- just use the cached one without downloading it again to check it has not changed on the server.
### how to configure?
Probably at the level of AnalysisActions.download_file(skip_redownload_verification=True) or something like that, and then mirrored in the rules that wrap it.
Contributor guide
Assessment
This issue has not been assessed yet.