bazelbuild / bazelbuild/bazel

Slow archive extraction using http_archive on Windows

Open
#20,264 2 comments 2 reactions 0 assignees View on GitHub
area-Windows P2 team-ExternalDeps type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

On windows when using http_archive with a large tar.gz or zip file, extraction can take minutes.

Manually extracting the same archive using 7zip takes seconds.

Temporarily disabling Windows Defender does not improve the timings.

### Which category does this issue belong to?

Performance

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Using llvm as a sample repo, create a WORKSPACE following this http_archive example:

https://github.com/llvm/llvm-project/blob/main/utils/bazel/examples/http_archive/WORKSPACE

Then run

` bazel build --generate_json_trace_profile --profile=build-profile.gz --experimental_announce_profile_path //...`

Looking at the profile, the extraction takes 1m 56s:

![image](https://github.com/bazelbuild/bazel/assets/4606363/75e742c0-616e-4a88-ade8-5cdc8e8e92d0)

Downloading the same archive and manually extracting with 7zip takes < 5s:

```
measure-command {7z x -y .\llvm-project-09ac97ce350316b95b8cddb796d52f71b6f68296.tar.gz}

Days : 0
Hours : 0
Minutes : 0
Seconds : 3
Milliseconds : 897
Ticks : 38976119
TotalDays : 4.51112488425926E-05
TotalHours : 0.00108266997222222
TotalMinutes : 0.0649601983333333
TotalSeconds : 3.8976119
TotalMilliseconds : 3897.6119
```

It appears we can't override the decompression library: https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/repository/CompressedTarFunction.java#L63

### Which operating system are you running Bazel on?

Windows 10.0.19045

### What is the output of `bazel info release`?

release 6.4.0

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

```text
fatal: not a git repository (or any of the parent directories): .git
```

### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

_No response_

### Have you found anything relevant by searching the web?

Note: my testing with Windows Defender off seemed to make no discernable improvement.

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with src/main/java/com/google/devtools/build/lib/bazel/repository/CompressedTarFunction.java, then reproduce the issue with the LLVM http_archive WORKSPACE example and the supplied Bazel profile command on Windows. Compare the archive extraction timing with the 7zip result and use the profile to identify the bottleneck; done means the Windows extraction is substantially faster without changing archive contents.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, operating-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.