ruby / ruby/zlib

`Zlib::GzipReader` doesn't read some large files

Open
#50 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
73
Forks
40
Avg merge
10h 32m
Merged PRs (30d)
2

Description

Hi.
I have to inflate a .csv.gz file which should return a 4 GB CSV with 25 million rows.

When I use an app or the gzip command line, I get the full file without issue.
When I use Zlib::GzipReader, only the first row is returned.

> Zlib::GzipReader.open("adresses-france.csv.gz") { |gz|  print gz.read }
id;id_fantoir;numero;rep;nom_voie;code_postal;code_insee;nom_commune;code_insee_ancienne_commune;nom_ancienne_commune;x;y;lon;lat;type_position;alias;nom_ld;libelle_acheminement;nom_afnor;source_position;source_nom_voie;certification_commune;cad_parcelles
 => nil

The file is provided by the french government:

There are many other files in the directory (for each region) but I cannot reproduce the issue with other files.

This service also provided a similar file in Addok format (https://adresse.data.gouv.fr/data/ban/adresses/latest/addok/adresses-addok-france.ndjson.gz) which should return a 3GB file with 2 million rows, but only the 25k first rows are returned by Zlib::GzipReader.

Is there any limit to what Zlib can support ? (size, rows, ..)
Does it come from the compressed file ?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the report with Zlib::GzipReader.open on adresses-france.csv.gz and compare its output with the gzip command-line result. Check whether the behavior is specific to the compressed file or to reading large files; done means identifying why the complete decompressed content is not returned.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.