fread should un-escape escaped quotes in fields
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 42/100
Research direction
Start by reproducing the issue with fread using the tbl1.csv and tbl2.csv examples in the report, then trace the fread parsing path for quoted fields. Compare the escaped-quote and doubled-quote cases. Done means fread reads tbl1.csv successfully and returns the same unescaped field values as read.table.
Written by the indexing model from the issue text.
Description
According to the docs:
A quoted field must start with quote and end with a quote that is also immediately followed by sep or \n. Thus, unescaped quotes may be present in a quoted field (
...,2,"Joe, "Bloggs"",3.14,...) as well as escaped quotes (...,2,"Joe \",Bloggs\"",3.14,...).
The following csv should be supported by fread.
library(data.table)
dt <- data.table(a = 1:2, b = c('f(c("a","b"))','sum(1,2)'))
dt
# a b
#1: 1 f(c("a","b"))
#2: 2 sum(1,2)
write.table(dt,"tbl1.csv",sep=",",na="",col.names=TRUE,row.names=FALSE,qmethod="escape")
write.table(dt,"tbl2.csv",sep=",",na="",col.names=TRUE,row.names=FALSE,qmethod="double")
system("cat tbl1.csv")
# "a","b"
#1,"f(c(\"a\",\"b\"))"
#2,"sum(1,2)"
system("cat tbl2.csv")
# "a","b"
#1,"f(c(""a"",""b""))"
#2,"sum(1,2)"
# output NA
fread("tbl1.csv",sep=",")
# Error in fread("tbl1.csv", sep = ",") :
# Expected sep (',') but new line, EOF (or other non printing character) ends field 1 when detecting types ( first): 2,"sum(1,2)"
# In addition: Warning message:
# In fread("tbl1.csv", sep = ",") :
# Starting data input on line 2 and discarded previous non-empty line: "a","b"
# incorrect output
fread("tbl2.csv",sep=",")
# a b
#1: 1 f(c(""a"",""b""))
#2: 2 sum(1,2)
# incorrect output
read.table("tbl1.csv",sep=",",header=TRUE)
# a b
#1 1 f(c(\\a\\,\\b\\))
#2 2 sum(1,2)
# correct output
read.table("tbl2.csv",sep=",",header=TRUE)
# a b
#1 1 f(c("a","b"))
#2 2 sum(1,2)
Findings:
as of now, only writing using qmethod="double" and read.table correctly supports write-read such kind of data.
Latest dev data.table, my locale, etc:
> sessionInfo()
# R version 3.1.3 (2015-03-09)
# Platform: x86_64-pc-linux-gnu (64-bit)
# Running under: Ubuntu 14.04.2 LTS
#
# locale:
# [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=en_DK.UTF-8 LC_COLLATE=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 LC_MESSAGES=C
# [7] LC_PAPER=en_US.UTF-8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#
# attached base packages:
# [1] stats graphics grDevices utils datasets methods base
#
# other attached packages:
# [1] data.table_1.9.5
#
# loaded via a namespace (and not attached):
# [1] bitops_1.0-6 chron_2.3-45 devtools_1.7.0 evaluate_0.5.5 formatR_1.0 httr_0.6.1 knitr_1.8 RCurl_1.95-4.5 stringr_0.6.2 tools_3.1.3
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·