fread gz runs out root drive space at / and may crash system
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 25/100
Research direction
Start at the fread() path that handles gzipped CSV input and reproduce the report with a large compressed file on a filesystem with limited root space. Done means reading the file no longer exhausts / or leaves truncated data, while preserving successful CSV import.
Written by the indexing model from the issue text.
Description
Create a csv of data of 100GB and gzip it (in my case it becomes 30G).
fread() the gzipped csv file will run out / and may crash the system sometimes even though my server has 1TB RAM and 256GB swap.
Filesystem Size Used Avail Use% Mounted on
udev 504G 0 504G 0% /dev
tmpfs 101G 2.4M 101G 1% /run
/dev/vg/root 81G 29G 48G 38% / ######
tmpfs 504G 2.4G 502G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 504G 0 504G 0% /sys/fs/cgroup
/dev/mapper/vg-home 799G 6.6G 751G 1% /home
/dev/mapper/hdd 55T 36T 17T 69% /auto
tmpfs 101G 0 101G 0% /run/user/1000
tmpfs 101G 0 101G 0% /run/user/1001
Filesystem Size Used Avail Use% Mounted on
udev 504G 0 504G 0% /dev
tmpfs 101G 2.4M 101G 1% /run
/dev/vg/root 81G 77G 0 100% / ######
tmpfs 504G 2.4G 502G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 504G 0 504G 0% /sys/fs/cgroup
/dev/mapper/vg-home 799G 6.6G 751G 1% /home
/dev/mapper/hdd 55T 36T 17T 69% /auto
tmpfs 101G 0 101G 0% /run/user/1000
tmpfs 101G 0 101G 0% /run/user/1001
And fread() will result in truncated data:
Avoidable 52.903 seconds. This file is very unusual: it ends abruptly without a final newline, and also its size is a multiple of 4096 bytes. Please properly end the last row with a newline using for example 'echo >> file' to avoid this time to copy.
|--------------------------------------------------|
|==================================================|
There were 50 or more warnings (use warnings() to see the first 50)
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·