fread and fwrite do not return identical results for gzipped file

Open
#5,239 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
r
Domain
data

Research direction

Start by reproducing the report with the linked 60MB CSV and the shown fread/fwrite calls on the reported Windows and R setup. Compare DT with DT_gzip using identical and fsetdiff, then inspect the fread and fwrite entry points. Done means the gzipped round trip reproduces the original data with identical(DT, DT_gzip) true and no row differences.

Written by the indexing model from the issue text.

Description

fread fwrite 
> library(data.table)
data.table 1.14.3 IN DEVELOPMENT built 2021-10-20 19:18:22 UTC; root using 6 threads (see ?getDTthreads).  Latest news: r-datatable.com
> DT <- fread("lcData100K.csv")
> fwrite(DT, "lcData100K.csv.gz")
> DT_gzip <- fread("lcData100K.csv.gz")
> identical(DT, DT_gzip)
[1] FALSE

File can be downloaded here (non-personal, so I can keep it there for a while). It's around 60MB, and I didn't quite have time to recreate a simpler example from scratch that could reproduce the issue. There are row level mismatches as well using fsetdiff:

> fsetdiff(DT_gzip, DT) |> nrow()
[1] 99966
> fsetdiff(DT, DT_gzip) |> nrow()
[1] 99966

I may be doing something stupidly wrong here, and if so, I'll close the issue. Ideally, fwrite and fread should reproduce the same file.

SessionInfo() 
> sessionInfo()
R version 4.1.1 (2021-08-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22000)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.14.3

loaded via a namespace (and not attached):
[1] compiler_4.1.1    tools_4.1.1       R.methodsS3_1.8.1 R.utils_2.11.0    R.oo_1.24.0
Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.