google / google/wireit

Restoring from cache is very slow for large files

Open
#605 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
6.4k
Forks
128
Avg merge
4d 10h
Merged PRs (30d)
4

Description

In our case, it takes around 2~5 minutes to just tar and untar all the cached files, which account for over 1GB in some cases.

[Tar is currently using `--gzip`](https://github.com/google/wireit/blob/32f805cb2ce82e24ca65f6abeb3cabaf8b64ea2b/src/caching/github-actions-cache.ts#L462) which is slow.
Since there are multiple fast alternatives to the plain gzip, including
- `--no-auto-compress` (No compression)
- `GZIP=--fast` ( -1 Fast flag)
- `Pigz` (Parallel Gzip)
- `Lzop`
- `Zstd`
each one with different tradeoffs.

So the easiest way to cope with different file types, size and use cases would be to provide a way to have the current `tar` flags set as default, and be able to override them through an environment variable or option.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.