apache / apache/beam

Provide a way to set non-default compression level when writing compressed files

Open
#20,560 2 comments 0 reactions 0 assignees View on GitHub
improvement io P3 python text
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

CompressedFile._initialize_compressor hardcodes the compression level used when writing:

 
self._compressor = zlib.compressobj(
          zlib.Z_DEFAULT_COMPRESSION, zlib.DEFLATED, self._gzip_mask)
 
It would be good to be able to control this, as I have a large set of GZIP compressed files that are creating output 10x larger then the input size when writing the same data back.
 
I've tried various monkeypatching approaches: these seem to work with the local runner, but failed when using DataflowRunner. For example:
 
class WriteData(beam.PTransform):
    def __init__(self, dst):
        import zlib

        self._dst = dst

        def _initialize_compressor(self):
            self._compressor = zlib.compressobj(
                zlib.Z_BEST_COMPRESSION, zlib.DEFLATED, self._gzip_mask
            )

        CompressedFile._initialize_compressor = _initialize_compressor

    def expand(self, p):
        return p | WriteToText(
            file_path_prefix=self._dst,
            file_name_suffix=".tsv.gz",
            compression_type="gzip",
        )

Imported from Jira [BEAM-11282](https://issues.apache.org/jira/browse/BEAM-11282). Original Jira may contain additional context.
Reported by: JackWhelpton.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.