Slow archive extraction using http_archive on Windows
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the bug:
On windows when using http_archive with a large tar.gz or zip file, extraction can take minutes.
Manually extracting the same archive using 7zip takes seconds.
Temporarily disabling Windows Defender does not improve the timings.
### Which category does this issue belong to?
Performance
### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Using llvm as a sample repo, create a WORKSPACE following this http_archive example:
https://github.com/llvm/llvm-project/blob/main/utils/bazel/examples/http_archive/WORKSPACE
Then run
` bazel build --generate_json_trace_profile --profile=build-profile.gz --experimental_announce_profile_path //...`
Looking at the profile, the extraction takes 1m 56s:

Downloading the same archive and manually extracting with 7zip takes < 5s:
```
measure-command {7z x -y .\llvm-project-09ac97ce350316b95b8cddb796d52f71b6f68296.tar.gz}
Days : 0
Hours : 0
Minutes : 0
Seconds : 3
Milliseconds : 897
Ticks : 38976119
TotalDays : 4.51112488425926E-05
TotalHours : 0.00108266997222222
TotalMinutes : 0.0649601983333333
TotalSeconds : 3.8976119
TotalMilliseconds : 3897.6119
```
It appears we can't override the decompression library: https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/repository/CompressedTarFunction.java#L63
### Which operating system are you running Bazel on?
Windows 10.0.19045
### What is the output of `bazel info release`?
release 6.4.0
### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.
_No response_
### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?
```text
fatal: not a git repository (or any of the parent directories): .git
```
### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.
_No response_
### Have you found anything relevant by searching the web?
Note: my testing with Windows Defender off seemed to make no discernable improvement.
### Any other information, logs, or outputs that you want to share?
_No response_
Contributor guide
Research direction
Start with src/main/java/com/google/devtools/build/lib/bazel/repository/CompressedTarFunction.java, then reproduce the issue with the LLVM http_archive WORKSPACE example and the supplied Bazel profile command on Windows. Compare the archive extraction timing with the 7zip result and use the profile to identify the bottleneck; done means the Windows extraction is substantially faster without changing archive contents.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, operating-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100