typelevel / typelevel/fs2

gzip compression not backwards compatible with gzip 1.3.12

Open
#1,883 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Scala
Stars
2.5k
Forks
636
Avg merge
2d 4h
Merged PRs (30d)
7

Description

Gzip compressed files produced with fs2.compression.gzip are not recognized by gzip 1.3.12. This is primarily an issue for users on CentOS 6 (CentOS 7 ships a newer version that handles this).

Here's how to reproduce:

Produce a compressed file:

import java.nio.file.Paths

import cats.effect.{Blocker, ExitCode, IO, IOApp}
import cats.syntax.all._

object WriteCompressedFile extends IOApp {
  override def run(args: List[String]): IO[ExitCode] = Blocker[IO].use { b =>
    fs2.Stream.emit("lorem ipsum")
      .covary[IO]
      .through(fs2.text.utf8Encode)
      .through(fs2.compression.gzip())
      .through(fs2.io.file.writeAll(Paths.get("/tmp/bar/foo.gz"), b)).compile.drain.as(ExitCode.Success)
  }
}

Uncompress on CentOS 6

├─ /tmp/bar
╰ file foo.gz
foo.gz: gzip compressed data, has CRC, from MacOS, original size modulo 2^32 37

├─ /tmp/bar
╰ docker run --rm -it -v $(pwd):/foo centos:centos6 bash
[root@52c59137bb4b /]# cd foo
[root@52c59137bb4b foo]# ls -l
total 4
-rw-r--r-- 1 root root 57 Jun  5 08:21 foo.gz
[root@52c59137bb4b foo]# file foo.gz
foo.gz: gzip compressed data, has CRC, from MacOS
[root@52c59137bb4b foo]# gunzip foo.gz
gzip: foo.gz is a a multi-part gzip file -- not supported
[root@52c59137bb4b foo]# gzip -d foo.gz
gzip: foo.gz is a a multi-part gzip file -- not supported
[root@52c59137bb4b foo]#

Related issue on CentOS:

Doesn't seem likely to change there (snipped from last link):

What we ship is 1.3.12 and will be for the lifespan of the distro. CentOS 7 uses 1.5.8 and will do for its life too.

Not sure if it's feasible to be backwards compatible with gzip à 2007, but it's good to document incompatibilities anyway.

Feel free to close if this is out of scope

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the fs2.compression.gzip entry point and reproduce the generated file with the Scala example, then test it with gzip 1.3.12 in the CentOS 6 environment described. Determine whether compatibility can be supported; if not, document the incompatibility and its affected versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.