fread may silently discard rows with only complete columns selected
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 45/100
Research direction
Start with the fread entry point and reproduce the five text examples in the issue, focusing on how trailing delimiters, missing fields, and select=1:2 affect row handling. Add regression coverage for these inputs and verify that no rows are silently discarded while the selected columns and warnings remain consistent.
Written by the indexing model from the issue text.
Description
I'm working with a csv-like data obtained from an online source (e.g. sina stock prices) where all except the last row end up with an extra comma. I replace all = with ,. Still, fread may not work properly with it.
The following are some examples to demonstrate the cases where fread may silently discard rows.
library(data.table)
fread(text = "a,10.0,0\nb,15.2,0\nc,10.3,0\n", header = FALSE, col.names = c("name", "value"), select = 1:2)
#> name value
#> 1: a 10.0
#> 2: b 15.2
#> 3: c 10.3
fread(text = "a,10.0,\nb,15.2,\nc,10.3,\n", header = FALSE, col.names = c("name", "value"), select = 1:2)
#> name value
#> 1: a 10.0
#> 2: b 15.2
#> 3: c 10.3
fread(text = "a,10.0,\nb,15.2,\nc,10.3\n", header = FALSE, col.names = c("name", "value"), select = 1:2)
#> Warning in fread(text = "a,10.0,\nb,15.2,\nc,10.3\n", header = FALSE,
#> col.names = c("name", : Discarded single-line footer: <<c,10.3>>
#> name value
#> 1: a 10.0
#> 2: b 15.2
fread(text = "a,10.0,\nb,15.2\nc,10.3\n", header = FALSE, col.names = c("name", "value"), select = 1:2)
#> name value
#> 1: b 15.2
#> 2: c 10.3
fread(text = "a,10.0,\nb,15.2\nc,10.3,\n", header = FALSE, col.names = c("name", "value"), select = 1:2)
#> name value
#> 1: c 10.3
fread(text = "a,10.0,\nb,15.2\n", header = FALSE, col.names = c("name", "value"), select = 1:2)
#> name value
#> 1: b 15.2
There seems to be obvious inconsistencies in the examples above.
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·