fread wrongly tries to load nested csv from properly quoted field & provides no option to work around this
Open
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
Research direction
Start with the reproducible R example using fwrite, fread, tempfile, and the iris data, focusing on how the quoted csv field is interpreted. Confirm the incorrect nested-CSV detection and determine the expected parsing or workaround behavior; done means the outer CSV is parsed correctly without treating its quoted field as a separate input.
Written by the indexing model from the issue text.
Description
fread
When parsing a .csv file with data.table where one of the columns is quoted (") and contains nested CSVs (which includes a sep and \n, but no nested "), it seems data.table incorrectly identifies the nested CSV as the data to read instead of parsing the outer, main CSV:
library(data.table)
tf1 <- tempfile()
tf2 <- tempfile()
fwrite(iris, tf1)
dt <- data.table(dataset = "iris", csv = paste0(readLines(tf1), collapse="\n"))
dt
#> dataset
#> <char>
#> 1: iris
#> csv
#> <char>
#> 1: Sepal.Length,Sepal.Width,Petal.Length,Petal.Width,Species\n5.1,3.5,1.4,0.2,setosa\n4.9,3,1.4,0.2,setosa\n4.7,3.2,1.3,0.2,setosa\n4.6,3.1,1.5,0.2,setosa\n5,3.6,1.4,0.2,setosa\n5.4,3.9,1.7,0.4,setosa\n4.6,3.4,1.4,0.3,setosa\n5,3.4,1.5,0.2,setosa\n4.4,2.9,1.4,0.2,setosa\n4.9,3.1,1.5,0.1,setosa\n5.4,3.7,1.5,0.2,setosa\n4.8,3.4,1.6,0.2,setosa\n4.8,3,1.4,0.1,setosa\n4.3,3,1.1,0.1,setosa\n5.8,4,1.2,0.2,setosa\n5.7,4.4,1.5,0.4,setosa\n5.4,3.9,1.3,0.4,setosa\n5.1,3.5,1.4,0.3,setosa\n5.7,3.8,1.7,0.3,setosa\n5.1,3.8,1.5,0.3,setosa\n5.4,3.4,1.7,0.2,setosa\n5.1,3.7,1.5,0.4,setosa\n4.6,3.6,1,0.2,setosa\n5.1,3.3,1.7,0.5,setosa\n4.8,3.4,1.9,0.2,setosa\n5,3,1.6,0.2,setosa\n5,3.4,1.6,0.4,setosa\n5.2,3.5,1.5,0.2,setosa\n5.2,3.4,1.4,0.2,setosa\n4.7,3.2,1.6,0.2,setosa\n4.8,3.1,1.6,0.2,setosa\n5.4,3.4,1.5,0.4,setosa\n5.2,4.1,1.5,0.1,setosa\n5.5,4.2,1.4,0.2,setosa\n4.9,3.1,1.5,0.2,setosa\n5,3.2,1.2,0.2,setosa\n5.5,3.5,1.3,0.2,setosa\n4.9,3.6,1.4,0.1,setosa\n4.4,3,1.3,0.2,setosa\n5.1,3.4,1.5,0.2,setosa\n5,3.5,1.3,0.3,setosa\n4.5,2.3,1.3,0.3,setosa\n4.4,3.2,1.3,0.2,setosa\n5,3.5,1.6,0.6,setosa\n5.1,3.8,1.9,0.4,setosa\n4.8,3,1.4,0.3,setosa\n5.1,3.8,1.6,0.2,setosa\n4.6,3.2,1.4,0.2,setosa\n5.3,3.7,1.5,0.2,setosa\n5,3.3,1.4,0.2,setosa\n7,3.2,4.7,1.4,versicolor\n6.4,3.2,4.5,1.5,versicolor\n6.9,3.1,4.9,1.5,versicolor\n5.5,2.3,4,1.3,versicolor\n6.5,2.8,4.6,1.5,versicolor\n5.7,2.8,4.5,1.3,versicolor\n6.3,3.3,4.7,1.6,versicolor\n4.9,2.4,3.3,1,versicolor\n6.6,2.9,4.6,1.3,versicolor\n5.2,2.7,3.9,1.4,versicolor\n5,2,3.5,1,versicolor\n5.9,3,4.2,1.5,versicolor\n6,2.2,4,1,versicolor\n6.1,2.9,4.7,1.4,versicolor\n5.6,2.9,3.6,1.3,versicolor\n6.7,3.1,4.4,1.4,versicolor\n5.6,3,4.5,1.5,versicolor\n5.8,2.7,4.1,1,versicolor\n6.2,2.2,4.5,1.5,versicolor\n5.6,2.5,3.9,1.1,versicolor\n5.9,3.2,4.8,1.8,versicolor\n6.1,2.8,4,1.3,versicolor\n6.3,2.5,4.9,1.5,versicolor\n6.1,2.8,4.7,1.2,versicolor\n6.4,2.9,4.3,1.3,versicolor\n6.6,3,4.4,1.4,versicolor\n6.8,2.8,4.8,1.4,versicolor\n6.7,3,5,1.7,versicolor\n6,2.9,4.5,1.5,versicolor\n5.7,2.6,3.5,1,versicolor\n5.5,2.4,3.8,1.1,versicolor\n5.5,2.4,3.7,1,versicolor\n5.8,2.7,3.9,1.2,versicolor\n6,2.7,5.1,1.6,versicolor\n5.4,3,4.5,1.5,versicolor\n6,3.4,4.5,1.6,versicolor\n6.7,3.1,4.7,1.5,versicolor\n6.3,2.3,4.4,1.3,versicolor\n5.6,3,4.1,1.3,versicolor\n5.5,2.5,4,1.3,versicolor\n5.5,2.6,4.4,1.2,versicolor\n6.1,3,4.6,1.4,versicolor\n5.8,2.6,4,1.2,versicolor\n5,2.3,3.3,1,versicolor\n5.6,2.7,4.2,1.3,versicolor\n5.7,3,4.2,1.2,versicolor\n5.7,2.9,4.2,1.3,versicolor\n6.2,2.9,4.3,1.3,versicolor\n5.1,2.5,3,1.1,versicolor\n5.7,2.8,4.1,1.3,versicolor\n6.3,3.3,6,2.5,virginica\n5.8,2.7,5.1,1.9,virginica\n7.1,3,5.9,2.1,virginica\n6.3,2.9,5.6,1.8,virginica\n6.5,3,5.8,2.2,virginica\n7.6,3,6.6,2.1,virginica\n4.9,2.5,4.5,1.7,virginica\n7.3,2.9,6.3,1.8,virginica\n6.7,2.5,5.8,1.8,virginica\n7.2,3.6,6.1,2.5,virginica\n6.5,3.2,5.1,2,virginica\n6.4,2.7,5.3,1.9,virginica\n6.8,3,5.5,2.1,virginica\n5.7,2.5,5,2,virginica\n5.8,2.8,5.1,2.4,virginica\n6.4,3.2,5.3,2.3,virginica\n6.5,3,5.5,1.8,virginica\n7.7,3.8,6.7,2.2,virginica\n7.7,2.6,6.9,2.3,virginica\n6,2.2,5,1.5,virginica\n6.9,3.2,5.7,2.3,virginica\n5.6,2.8,4.9,2,virginica\n7.7,2.8,6.7,2,virginica\n6.3,2.7,4.9,1.8,virginica\n6.7,3.3,5.7,2.1,virginica\n7.2,3.2,6,1.8,virginica\n6.2,2.8,4.8,1.8,virginica\n6.1,3,4.9,1.8,virginica\n6.4,2.8,5.6,2.1,virginica\n7.2,3,5.8,1.6,virginica\n7.4,2.8,6.1,1.9,virginica\n7.9,3.8,6.4,2,virginica\n6.4,2.8,5.6,2.2,virginica\n6.3,2.8,5.1,1.5,virginica\n6.1,2.6,5.6,1.4,virginica\n7.7,3,6.1,2.3,virginica\n6.3,3.4,5.6,2.4,virginica\n6.4,3.1,5.5,1.8,virginica\n6,3,4.8,1.8,virginica\n6.9,3.1,5.4,2.1,virginica\n6.7,3.1,5.6,2.4,virginica\n6.9,3.1,5.1,2.3,virginica\n5.8,2.7,5.1,1.9,virginica\n6.8,3.2,5.9,2.3,virginica\n6.7,3.3,5.7,2.5,virginica\n6.7,3,5.2,2.3,virginica\n6.3,2.5,5,1.9,virginica\n6.5,3,5.2,2,virginica\n6.2,3.4,5.4,2.3,virginica\n5.9,3,5.1,1.8,virginica
fwrite(dt, tf2)
fread(tf2, verbose = TRUE) # bad output +warning
#> OpenMP version (_OPENMP) 201511
#> omp_get_num_procs() 128
#> R_DATATABLE_NUM_PROCS_PERCENT unset (default 50)
#> R_DATATABLE_NUM_THREADS unset
#> R_DATATABLE_THROTTLE unset (default 1024)
#> omp_get_thread_limit() 16
#> omp_get_max_threads() 128
#> OMP_THREAD_LIMIT 16
#> OMP_NUM_THREADS unset
#> RestoreAfterFork true
#> data.table is using 16 threads with throttle==1024. See ?setDTthreads.
#> freadR.c has been passed a filename: /tmp/RtmpdDWffF/file208e3aff7a42
#> [01] Check arguments
#> Using 16 threads (omp_get_max_threads()=128, nth=16)
#> NAstrings = [<<NA>>]
#> None of the NAstrings look like numbers.
#> show progress = 0
#> 0/1 column will be read as integer
#> [02] Opening the file
#> Opening file /tmp/RtmpdDWffF/file208e3aff7a42
#> File opened, size = 3.647KB (3735 bytes).
#> Memory mapped ok
#> [03] Detect and skip BOM
#> [04] Arrange mmap to be \0 terminated
#> \n has been found in the input and different lines can end with different line endings (e.g. mixed \n and \r\n in one file). This is common and ideal.
#> [05] Skipping initial rows if needed
#> Positioned on line 1 starting: <<dataset,csv>>
#> [06] Detect separator, quoting rule, and ncolumns
#> Detecting sep automatically ...
#> sep=',' with 2 lines of 2 fields using quote rule 0
#> sep=',' with 98 lines of 5 fields using quote rule 2
#> Warning in fread(tf2, verbose = TRUE): Found and resolved improper quoting in
#> first 100 rows. If the fields are not quoted (e.g. field separator does not
#> appear within any field), try quote="" to avoid this warning.
#> Detected 5 columns on line 3. This line is either column names or first data row. Line starts as: <<5.1,3.5,1.4,0.2,setosa>>
#> Quote rule picked = 2
#> fill=false and the most number of columns found is 5
#> [07] Detect column types, good nrow estimate and whether first row is column names
#> Number of sampling jump points = 1 because (3658 bytes from row 1 to eof) / (2 * 2356 jump0size) == 0
#> Type codes (jump 000) : 8888D Quote rule 2
#> Type codes (jump 001) : 8888D Quote rule 2
#> Types in 1st data row match types in 2nd data row but previous row has 6 fields. Taking previous row as column names.
#> Warning in fread(tf2, verbose = TRUE): Detected 6 column names but the data has
#> 5 columns. Filling rows automatically. Set fill=TRUE explicitly to avoid this
#> warning.
#> =====
#> Sampled 147 rows (handled \n inside quoted fields) at 2 jump points
#> Bytes from first data row on line 3 to the end of last row: 3635
#> Line length: mean=24.39 sd=1.97 min=19 max=27
#> Estimated number of rows: 3635 / 24.39 = 150
#> Initial alloc = 177 rows (150 + 18%) using bytes/max(mean-2*sd,min) clamped between [1.1*estn, 2.0*estn]
#> =====
#> [08] Assign column names
#> [09] Apply user overrides on column types
#> After 0 type and 0 drop user overrides : 8888D1
#> [10] Allocate memory for the datatable
#> Allocating 6 column slots (6 - 0 dropped) with 177 rows
#> [11] Read the data
#> jumps=[0..1), chunk_size=1048576, total_size=3658
#> Read 150 rows x 6 columns from 3.647KB (3735 bytes) file in 00:00.001 wall clock time
#> [12] Finalizing the datatable
#> Type counts:
#> 1 : bool8 '1'
#> 4 : float64 '8'
#> 1 : string 'D'
#> =============================
#> 0.000s ( 7%) Memory map 0.000GB file
#> 0.001s ( 85%) sep=',' ncol=6 and header detection
#> 0.000s ( 1%) Column type detection using 147 sample rows
#> 0.000s ( 1%) Allocation of 177 rows x 6 cols (0.000GB) of which 150 ( 85%) rows used
#> 0.000s ( 6%) Reading 1 chunks (0 swept) of 1.000MB (each chunk 150 rows) using 1 threads
#> + 0.000s ( 3%) Parse to row-major thread buffers (grown 0 times)
#> + 0.000s ( 1%) Transpose
#> + 0.000s ( 2%) Waiting
#> 0.000s ( 0%) Rereading 0 columns due to out-of-sample type exceptions
#> 0.001s Total
#> iris "Sepal.Length Sepal.Width Petal.Length Petal.Width Species
#> <num> <num> <num> <num> <char> <lgcl>
#> 1: 5.1 3.5 1.4 0.2 setosa NA
#> 2: 4.9 3.0 1.4 0.2 setosa NA
#> 3: 4.7 3.2 1.3 0.2 setosa NA
#> 4: 4.6 3.1 1.5 0.2 setosa NA
#> 5: 5.0 3.6 1.4 0.2 setosa NA
#> ---
#> 146: 6.7 3.0 5.2 2.3 virginica NA
#> 147: 6.3 2.5 5.0 1.9 virginica NA
#> 148: 6.5 3.0 5.2 2.0 virginica NA
#> 149: 6.2 3.4 5.4 2.3 virginica NA
#> 150: 5.9 3.0 5.1 1.8 virginica" NA
fread(tf2, nrow=1) # correct output
#> dataset
#> <char>
#> 1: iris
#> csv
#> <char>
#> 1: Sepal.Length,Sepal.Width,Petal.Length,Petal.Width,Species\n5.1,3.5,1.4,0.2,setosa\n4.9,3,1.4,0.2,setosa\n4.7,3.2,1.3,0.2,setosa\n4.6,3.1,1.5,0.2,setosa\n5,3.6,1.4,0.2,setosa\n5.4,3.9,1.7,0.4,setosa\n4.6,3.4,1.4,0.3,setosa\n5,3.4,1.5,0.2,setosa\n4.4,2.9,1.4,0.2,setosa\n4.9,3.1,1.5,0.1,setosa\n5.4,3.7,1.5,0.2,setosa\n4.8,3.4,1.6,0.2,setosa\n4.8,3,1.4,0.1,setosa\n4.3,3,1.1,0.1,setosa\n5.8,4,1.2,0.2,setosa\n5.7,4.4,1.5,0.4,setosa\n5.4,3.9,1.3,0.4,setosa\n5.1,3.5,1.4,0.3,setosa\n5.7,3.8,1.7,0.3,setosa\n5.1,3.8,1.5,0.3,setosa\n5.4,3.4,1.7,0.2,setosa\n5.1,3.7,1.5,0.4,setosa\n4.6,3.6,1,0.2,setosa\n5.1,3.3,1.7,0.5,setosa\n4.8,3.4,1.9,0.2,setosa\n5,3,1.6,0.2,setosa\n5,3.4,1.6,0.4,setosa\n5.2,3.5,1.5,0.2,setosa\n5.2,3.4,1.4,0.2,setosa\n4.7,3.2,1.6,0.2,setosa\n4.8,3.1,1.6,0.2,setosa\n5.4,3.4,1.5,0.4,setosa\n5.2,4.1,1.5,0.1,setosa\n5.5,4.2,1.4,0.2,setosa\n4.9,3.1,1.5,0.2,setosa\n5,3.2,1.2,0.2,setosa\n5.5,3.5,1.3,0.2,setosa\n4.9,3.6,1.4,0.1,setosa\n4.4,3,1.3,0.2,setosa\n5.1,3.4,1.5,0.2,setosa\n5,3.5,1.3,0.3,setosa\n4.5,2.3,1.3,0.3,setosa\n4.4,3.2,1.3,0.2,setosa\n5,3.5,1.6,0.6,setosa\n5.1,3.8,1.9,0.4,setosa\n4.8,3,1.4,0.3,setosa\n5.1,3.8,1.6,0.2,setosa\n4.6,3.2,1.4,0.2,setosa\n5.3,3.7,1.5,0.2,setosa\n5,3.3,1.4,0.2,setosa\n7,3.2,4.7,1.4,versicolor\n6.4,3.2,4.5,1.5,versicolor\n6.9,3.1,4.9,1.5,versicolor\n5.5,2.3,4,1.3,versicolor\n6.5,2.8,4.6,1.5,versicolor\n5.7,2.8,4.5,1.3,versicolor\n6.3,3.3,4.7,1.6,versicolor\n4.9,2.4,3.3,1,versicolor\n6.6,2.9,4.6,1.3,versicolor\n5.2,2.7,3.9,1.4,versicolor\n5,2,3.5,1,versicolor\n5.9,3,4.2,1.5,versicolor\n6,2.2,4,1,versicolor\n6.1,2.9,4.7,1.4,versicolor\n5.6,2.9,3.6,1.3,versicolor\n6.7,3.1,4.4,1.4,versicolor\n5.6,3,4.5,1.5,versicolor\n5.8,2.7,4.1,1,versicolor\n6.2,2.2,4.5,1.5,versicolor\n5.6,2.5,3.9,1.1,versicolor\n5.9,3.2,4.8,1.8,versicolor\n6.1,2.8,4,1.3,versicolor\n6.3,2.5,4.9,1.5,versicolor\n6.1,2.8,4.7,1.2,versicolor\n6.4,2.9,4.3,1.3,versicolor\n6.6,3,4.4,1.4,versicolor\n6.8,2.8,4.8,1.4,versicolor\n6.7,3,5,1.7,versicolor\n6,2.9,4.5,1.5,versicolor\n5.7,2.6,3.5,1,versicolor\n5.5,2.4,3.8,1.1,versicolor\n5.5,2.4,3.7,1,versicolor\n5.8,2.7,3.9,1.2,versicolor\n6,2.7,5.1,1.6,versicolor\n5.4,3,4.5,1.5,versicolor\n6,3.4,4.5,1.6,versicolor\n6.7,3.1,4.7,1.5,versicolor\n6.3,2.3,4.4,1.3,versicolor\n5.6,3,4.1,1.3,versicolor\n5.5,2.5,4,1.3,versicolor\n5.5,2.6,4.4,1.2,versicolor\n6.1,3,4.6,1.4,versicolor\n5.8,2.6,4,1.2,versicolor\n5,2.3,3.3,1,versicolor\n5.6,2.7,4.2,1.3,versicolor\n5.7,3,4.2,1.2,versicolor\n5.7,2.9,4.2,1.3,versicolor\n6.2,2.9,4.3,1.3,versicolor\n5.1,2.5,3,1.1,versicolor\n5.7,2.8,4.1,1.3,versicolor\n6.3,3.3,6,2.5,virginica\n5.8,2.7,5.1,1.9,virginica\n7.1,3,5.9,2.1,virginica\n6.3,2.9,5.6,1.8,virginica\n6.5,3,5.8,2.2,virginica\n7.6,3,6.6,2.1,virginica\n4.9,2.5,4.5,1.7,virginica\n7.3,2.9,6.3,1.8,virginica\n6.7,2.5,5.8,1.8,virginica\n7.2,3.6,6.1,2.5,virginica\n6.5,3.2,5.1,2,virginica\n6.4,2.7,5.3,1.9,virginica\n6.8,3,5.5,2.1,virginica\n5.7,2.5,5,2,virginica\n5.8,2.8,5.1,2.4,virginica\n6.4,3.2,5.3,2.3,virginica\n6.5,3,5.5,1.8,virginica\n7.7,3.8,6.7,2.2,virginica\n7.7,2.6,6.9,2.3,virginica\n6,2.2,5,1.5,virginica\n6.9,3.2,5.7,2.3,virginica\n5.6,2.8,4.9,2,virginica\n7.7,2.8,6.7,2,virginica\n6.3,2.7,4.9,1.8,virginica\n6.7,3.3,5.7,2.1,virginica\n7.2,3.2,6,1.8,virginica\n6.2,2.8,4.8,1.8,virginica\n6.1,3,4.9,1.8,virginica\n6.4,2.8,5.6,2.1,virginica\n7.2,3,5.8,1.6,virginica\n7.4,2.8,6.1,1.9,virginica\n7.9,3.8,6.4,2,virginica\n6.4,2.8,5.6,2.2,virginica\n6.3,2.8,5.1,1.5,virginica\n6.1,2.6,5.6,1.4,virginica\n7.7,3,6.1,2.3,virginica\n6.3,3.4,5.6,2.4,virginica\n6.4,3.1,5.5,1.8,virginica\n6,3,4.8,1.8,virginica\n6.9,3.1,5.4,2.1,virginica\n6.7,3.1,5.6,2.4,virginica\n6.9,3.1,5.1,2.3,virginica\n5.8,2.7,5.1,1.9,virginica\n6.8,3.2,5.9,2.3,virginica\n6.7,3.3,5.7,2.5,virginica\n6.7,3,5.2,2.3,virginica\n6.3,2.5,5,1.9,virginica\n6.5,3,5.2,2,virginica\n6.2,3.4,5.4,2.3,virginica\n5.9,3,5.1,1.8,virginica
Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.1.0 (2021-05-18)
#> os CentOS Linux 7 (Core)
#> system x86_64, linux-gnu
#> ui X11
#> language en_US.UTF-8
#> collate en_US.UTF-8
#> ctype en_US.UTF-8
#> tz Europe/Berlin
#> date 2023-07-13
#> pandoc 2.19.2 @ /usr/lib/rstudio-server/bin/quarto/bin/tools/ (via rmarkdown)
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> package * version date (UTC) lib source
#> cli 3.6.1 2023-03-23 [1] CRAN (R 4.1.0)
#> data.table * 1.14.9 2023-07-13 [1] local
#> digest 0.6.29 2021-12-01 [1] CRAN (R 4.1.0)
#> evaluate 0.14 2019-05-28 [1] CRAN (R 4.1.0)
#> fastmap 1.1.0 2021-01-25 [1] CRAN (R 4.1.0)
#> fs 1.5.2 2021-12-08 [1] CRAN (R 4.1.0)
#> glue 1.6.2 2022-02-24 [1] CRAN (R 4.1.0)
#> highr 0.9 2021-04-16 [1] CRAN (R 4.1.0)
#> htmltools 0.5.5 2023-03-23 [1] CRAN (R 4.1.0)
#> knitr 1.37 2021-12-16 [1] CRAN (R 4.1.0)
#> lifecycle 1.0.3 2022-10-07 [1] CRAN (R 4.1.0)
#> magrittr 2.0.2 2022-01-26 [1] CRAN (R 4.1.0)
#> purrr 1.0.1 2023-01-10 [1] CRAN (R 4.1.0)
#> R.cache 0.16.0 2022-07-21 [1] CRAN (R 4.1.0)
#> R.methodsS3 1.8.1 2020-08-26 [1] CRAN (R 4.1.0)
#> R.oo 1.24.0 2020-08-26 [1] CRAN (R 4.1.0)
#> R.utils 2.11.0 2021-09-26 [1] CRAN (R 4.1.0)
#> reprex 2.0.1 2021-08-05 [1] CRAN (R 4.1.0)
#> rlang 1.1.1 2023-04-28 [1] CRAN (R 4.1.0)
#> rmarkdown 2.23 2023-07-01 [1] CRAN (R 4.1.0)
#> rstudioapi 0.13 2020-11-12 [1] CRAN (R 4.1.0)
#> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.1.0)
#> stringi 1.7.6 2021-11-29 [1] CRAN (R 4.1.0)
#> stringr 1.5.0 2022-12-02 [1] CRAN (R 4.1.0)
#> styler 1.10.1 2023-06-05 [1] CRAN (R 4.1.0)
#> vctrs 0.6.2 2023-04-19 [1] CRAN (R 4.1.0)
#> withr 2.5.0 2022-03-03 [1] CRAN (R 4.1.0)
#> xfun 0.39 2023-04-20 [1] CRAN (R 4.1.0)
#> yaml 2.3.4 2022-02-17 [1] CRAN (R 4.1.0)
#>
#> [1] /home/gleixner/R/x86_64-pc-linux-gnu-library/4.1.0d
#> [2] /software/r/4.1.0/lib64/R/library
#>
#> ──────────────────────────────────────────────────────────────────────────────
In above example, specifying nrow=1 allows data.table to correctly interpret and read the CSV but usually the correct number of rows is not know in advance.
There does not appear to be any provided means in data.table to turn off the automatic guessing of sep, quoting, and nrow. I've tried manually specifying sep="," and/or quoting and skip=0, but to no avail.
A potential improvement could be an option to disable this automatic guessing feature. Alternatively, or additionally, data.table could provide a fallback mechanism that tries alternative guesses if the initial selected guess fails. Another proposal would be to prioritize guesses by number of characters instead of number of fields (I believe the latter to be the current strategy).
For context, the "guesses" I am referring to are:
#> Detecting sep automatically ...
#> sep=',' with 2 lines of 2 fields using quote rule 0
#> sep=',' with 98 lines of 5 fields using quote rule 2
PS: I really hope I did not miss the original issue if this is a duplicate.
- Dominant language
- R
- Stars
- 3.9k
- Forks
- 1.1k
- Avg merge
- 14h 4m
- Merged PRs (30d)
- 4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Rdatatable/data.table
-
as.data.table() recurses without end on a survival::Surv object (or any data.frame carrying one) Open
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
Rdatatable/data.table#7887 ·
-
consistency tests
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#7853 · 3 comments ·
-
internals
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#6938 · 1 comment ·
-
encoding fread
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Rdatatable/data.table#5179 · 8 comments ·
-
documentation programming
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Rdatatable/data.table#3199 · 3 comments ·
All issues in Rdatatable/data.table
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
r-lib/pkgdepends#485 · 3 comments ·
-
Difficulty 1/5 Under an hour Newbie friendliness 92/100
-
beginners blocker
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
enviPathR OpenBuild Error Build OK Build Warning policies-accepted pre-review precheck-passed
Difficulty 1/5 Under an hour Newbie friendliness 84/100
Bioconductor/BiocContributions#207 · 6 comments ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
datacarpentry/semester-biology#1255 ·