Add option to bazel fetch to disable extraction of the downloaded tarballs
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 75
Description
### Description of the feature request:
Add an option to `bazel fetch` that disables extractions of the tarballs (e.g. `--no-extract` or `--download-only` or something like that).
### Feature requests: what underlying problem are you trying to solve with this feature?
I'm trying to build TensorFlow with Bazel on an offline system. Based on https://docs.bazel.build/versions/master/external.html#offline-builds and https://docs.bazel.build/versions/master/guide.html#running-bazel-in-an-airgapped-environment I find that I can do this by running my `bazel build` on the offline systems using the `--distdir` argument, and pointing that `distdir` to a directory with all the tarballs.
I've run `bazel fetch` on an online system, in the hope that this would get me the tarballs that I could ship to the offline system. Unfortunately, `bazel fetch` seems to store the tarballs only temporarily and immediately extracts them. Shipping the extracted folders to the offline system and pointing the `--distdir` to it doesn't work (since it searches distdir for tarballs with the right checksum according to https://docs.bazel.build/versions/master/guide.html#distdir , not for the extracted folders).
Of course, I could download all external packages/tarballs manually, but the TensorFlow build seems to require over 200 of them - not something you want to do by hand. I've tried this with one or two, to check that the solution with passing a `--distdir` on the offline system indeed works, and that seems to be the case (it no longer complains about the 2 packages I downloaded and shipped manually to my offline system).
All in all, I would envision supporting the following procedure that would better support offline builds:
On the online system:
```
bazel --output_user_root=/my/tarball/dir fetch --no-extract
# Some way to ship to the offline system, in my case my online system is whitelisted:
scp -r /my/tarball/dir user@offline_system:/my/dist/dir
```
Then, on the offline system, build with:
```
bazel build --distdir /my/dist/dir
```
Of course, it is entirely possible that I have misunderstood the documentation and that there is already a (simple) possibility to get the tarballs for offline builds. If that's the case, I'd really appreciate being pointed to the relevant docs.
### What operating system are you running Bazel on?
Online system: Debian 10
Offline system: CentOS 8
### What's the output of `bazel info release`?
3.7.2 (the full output is very long, but not so relevant for feature request I suppose)
### Have you found anything relevant by searching the web?
https://docs.bazel.build/versions/master/external.html#offline-builds
https://docs.bazel.build/versions/master/guide.html#running-bazel-in-an-airgapped-environment
https://stackoverflow.com/questions/61820038/dependency-archiving-for-airgapped-network
Contributor guide
Research direction
Start at the bazel fetch command and read the linked offline-build and airgapped-environment documentation to understand how downloads and extraction currently interact with --distdir. Done means a fetch option can retain the downloaded tarballs without extracting them, so they can be copied and used by an offline bazel build.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100