bazelbuild / bazelbuild/rules_pkg

Add attribute to set compression type and level for pkg_tar

Open
#60 3 comments 7 reactions 0 assignees View on GitHub
good first issue help wanted P3
Dominant language
Starlark
Stars
253
Forks
221
Avg merge
10h 33m
Merged PRs (30d)
1

Description

### Description of the problem / feature request:

pkg_tar does not have an attribute that exposes the compresslevel argument for tarfile. The [default value](https://docs.python.org/2.7/library/tarfile.html#tarfile.open) for this argument is 9, which is max compression, slowest speed. This can take twice as much time.

If we don't want to have an extra attribute, then at least the default value should be 6 -- the same as in [gzip CLI](https://www.gnu.org/software/gzip/manual/gzip.html).

https://github.com/bazelbuild/bazel/blob/0b6899be05d51088267ce213850b9ead96d68b8e/tools/build_defs/pkg/archive.py#L119

### Feature requests: what underlying problem are you trying to solve with this feature?

Make pkg_tar faster for larger archives when using compression.

```
$ python --version
Python 2.7.12
$ docker pull rocker/r-ver:3.4.4
$ docker save rocker/r-ver:3.4.4 -o image.tar
$ cat test.py
import sys
import gzip
import shutil

with open('image.tar', 'rb') as f_in, gzip.open('image.tar.gz.' + sys.argv[1], 'wb', compresslevel=int(sys.argv[1])) as f_out:
shutil.copyfileobj(f_in, f_out)
$ time python test.py 1
real 0m12.320s ...
$ time python test.py 6
real 0m28.616s ...
$ time python test.py 9
real 1m37.240s ...
$ ls -lh image.tar.gz.*
... 237M ... image.tar.gz.1
... 217M ... image.tar.gz.6
... 216M ... image.tar.gz.9
$ time gzip -1 -k image.tar
real 0m11.855s ...
$ time gzip -6 -k image.tar
real 0m27.176s ...
$ time gzip -9 -k image.tar
real 1m32.707s ...
```

### What operating system are you running Bazel on?

Ubuntu

### What's the output of `bazel info release`?

release 0.9.0

### Have you found anything relevant by searching the web?

There is an [opinion](https://github.com/google/containerregistry/issues/26#issuecomment-336544905) in many places that pkg_tar, or other python based archiving tools in bazel repos, are slow because python's gzip is slow. But that is not my experience in the example above.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.