Numeric variable parsed as character when using comma decimals and separator in fread
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
Research direction
Start by reproducing the supplied quote1–quote4 examples through data.table::fread, focusing on type inference when the first value is missing, comma decimals are enabled, and separators vary. Trace the fread parsing path and verify that the affected var1 column is read as numeric with the expected missing value and decimal conversion.
Written by the indexing model from the issue text.
Description
Quite new to fread and data.table.
The problem occurred when I had a dataset like quote1, which lead to var1 being interpreted as character, instead of numeric.
Managed to isolate the issue in the following examples. There seem to be some kind of parsing issue when the first value is missing and it is followed by a date. It also seems related to what separator is used.
quote1 <- 'var1,var2\n,2014-06-15\n"15,3",2014-06-16'
data.table::fread(quote1, dec = ',', header = TRUE)
# quote 1 output
> data.table::fread(quote1, dec = ',', header = TRUE)
var1 var2
1: 2014-06-15
2: 15,3 2014-06-16
quote2 <- 'var1,var2\n"15,3",2014-06-15\n"15,3",2014-06-16'
data.table::fread(quote2, dec = ',', header = TRUE)
# quote 2 output
> data.table::fread(quote2, dec = ',', header = TRUE)
var1 var2
1: 15.3 2014-06-15
2: 15.3 2014-06-16
quote3 <- 'var1,var2\n,"2014-06-15"\n"15,3","2014-06-16"'
data.table::fread(quote3, dec = ',', header = TRUE)
# quote 3 output
> data.table::fread(quote3, dec = ',', header = TRUE)
var1 var2
1: NA 2014-06-15
2: 15.3 2014-06-16
quote4 <- 'var1;var2\n;2014-06-15\n"15,3";2014-06-16'
data.table::fread(quote4, dec = ',', header = TRUE, sep = ';')
# quote 4 output
> data.table::fread(quote4, dec = ',', header = TRUE, sep = ';')
var1 var2
1: NA 2014-06-15
2: 15.3 2014-06-16
Trying to use colClasses does not work either:
data.table::fread(quote1, dec = ',', header = TRUE, colClasses = c("var1" = "numeric"))
# output
var1 var2
1: 2014-06-15
2: 15,3 2014-06-16
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·