aboutcode-org / aboutcode-org/scancode-toolkit

extractcode fails to unpack some gzipped files where the only issue is an extra trailing newline

Open
#2,740 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.6k
Forks
791
Avg merge
1d 12h
Merged PRs (30d)
5

Description

### Description

This case was originally proposed in the discussion of #2723.

There are some common conditions where extractcode fails to recursively unpack an archive, and one of those "fairly common" conditions is where a gzipped file happens to contain a trailing newline after the archive. In truth, this makes the gzipped file an invalid archive, so gunzip fails to unpack it and returns an error. However, other tools (such as zless) will display the uncorrupted portions of the archive, and this allows people to view the content. If this content, viewable by a common viewer, contains license or copyright references, it will be missed by the extractcode/scancode combination.

As a proposal subject to discussion, perhaps extractcode could produce a warning in this case, while extracting the uncorrupted portion of the file by using a streaming-style of unpack (zless or equivalent). Whether this extra code complexity is justified, and how to determine whether the streaming unpack produced something useful needs to be discussed. An alternate implementation might be to truncate the original by one byte and try unpack again. If this succeeds, then silently move on.

### How To Reproduce

The target archive is [here](https://github.com/ruby/rake/archive/refs/tags/v0.9.2.2.zip). Within the Ruby Rake zipfile is what appears to be a compressed documentation file, rake-0.9.2.2/doc/rake.1.gz which could be directly downloaded [here](https://github.com/ruby/rake/blob/v0.9.2.2/doc/rake.1.gz) if you wish.

Manually trying to unzip the offending rake.1.gz file, I get:

gunzip rake.1.gz
gzip: rake.1.gz: unexpected end of file

echo $?
1

### System configuration

My test system is: Debian Linux, python 3.7.3, scancode-toolkit 30.1.0, installed from download of released Linux tar archive

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.