fread: need more flexible behavior when encountering a broken line.
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
Research direction
Start at the fread entry point and trace the existing fill=TRUE handling for lines with inconsistent field counts. Define coverage for the proposed error, fill, skip, and extract strategies, including report output and the extract fields lineno, rowno, line, and nfields; done means each mode has tested, consistent behavior.
Written by the indexing model from the issue text.
Description
A new parameter bad.lines (or similar) is proposed. This parameter adjusts fread's strategy when dealing with lines that are "broken" (i.e. have less or more than the required number of columns). This parameter may take the following values:
"error"(default) -- stop scanning the file and raise an exception."fill"(currently achieved withfill=TRUE) -- any lines having too few fields are padded with NAs. Here "too few" means less than the maximum number of fields observed across all rows in the file."skip"-- broken lines are simply ignored."extract"-- any broken lines are placed into a separate datatable, whereas the "main" datatable retains empty rows in their place. The extra datatable will have at least the following fields:lineno(line number in the original data file),rowno(corresponding row number in the "main" datatable),line(the text of the line),nfields(number of fields detected on that line)
Additionally, there should be parameter report (default FALSE), which is used for strategies "fill" and "skip", and instructs fread to report to the user line numbers that were filled/skipped.
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·