Assumption of sep=',' is too strong in some files with dec=','

Open
#6,447 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
30/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Stale
Tech stack
r
Domain
data

Research direction

Reproduce the reported fread(text=...) example with verbose output and trace the separator and header-detection path that chooses sep=','. Investigate how a single-column timestamp containing a comma should be recognized, then add coverage for the intended behavior and confirm the warning and parsed columns are correct.

Written by the indexing model from the issue text.

Description

fread

As observed working on #6440:

fread(text="t\n2023-10-12T06:53:53,123Z", verbose=TRUE)

with output:

  OpenMP version (_OPENMP)       201511
  omp_get_num_procs()            2
  R_DATATABLE_NUM_PROCS_PERCENT  unset (default 50)
  R_DATATABLE_NUM_THREADS        unset
  R_DATATABLE_THROTTLE           unset (default 1024)
  omp_get_thread_limit()         2147483647
  omp_get_max_threads()          2
  OMP_THREAD_LIMIT               unset
  OMP_NUM_THREADS                unset
  RestoreAfterFork               true
  data.table is using 1 threads with throttle==1024. See ?setDTthreads.
freadR.c has been passed the data as text input (not a filename)
[01] Check arguments
  Using 1 threads (omp_get_max_threads()=2, nth=1)
  NAstrings = [<<NA>>]
  None of the NAstrings look like numbers.
  show progress = 1
  0/1 column will be read as integer
[02] Opening the file
  `input` argument is provided rather than a file name, interpreting as raw text to read
[03] Detect and skip BOM
[04] Arrange mmap to be \0 terminated
  \n has been found in the input and different lines can end with different line endings (e.g. mixed \n and \r\n in one file). This is common and ideal.
[05] Skipping initial rows if needed
  Positioned on line 1 starting: <<t>>
[06] Detect separator, quoting rule, and ncolumns
  Detecting sep automatically ...
  sep=','  with 1 lines of 2 fields using quote rule 0
  Detected 2 columns on line 1. This line is either column names or first data row. Line starts as: <<2023-10-12T06:53:53,123Z>>
  Quote rule picked = 0
  fill=false and the most number of columns found is 2
[07] Detect column types, dec, good nrow estimate and whether first row is column names
  sep=',' so dec set to '.'
  Number of sampling jump points = 1 because (24 bytes from row 1 to eof) / (2 * 24 jump0size) == 0
Types in 1st data row match types in 2nd data row but previous row has 1 fields. Taking previous row as column names.  All rows were sampled since file is small so we know nrow=0 exactly
[08] Assign column names
[09] Apply user overrides on column types
  After 0 type and 0 drop user overrides : 11
[10] Allocate memory for the datatable
  Allocating 2 column slots (2 - 0 dropped) with 0 rows
[11] Read the data
  jumps=[0..1), chunk_size=1048576, total_size=24
  Too few rows allocated. Allocating additional 1024 rows (now nrows=1024) and continue reading from jump 0
  jumps=[0..1), chunk_size=1048576, total_size=24
  2 out-of-sample type bumps: CD
  jumps=[0..1), chunk_size=1048576, total_size=24
Read 1 rows x 2 columns from 26 bytes file in 00:00.001 wall clock time
[12] Finalizing the datatable
  Type counts:
         1 : float64   'C'
         1 : string    'D'
=============================
   0.000s ( 16%) Memory map 0.000GB file
   0.000s ( 50%) sep=',' ncol=2 and header detection
   0.000s (  3%) Column type detection using 0 sample rows
   0.000s (  3%) Allocation of 1024 rows x 2 cols (0.000GB) of which 1 (  0%) rows used
   0.000s ( 28%) Reading 1 chunks (0 swept) of 1.000MB (each chunk 1 rows) using 1 threads
   +    0.000s (  1%) Parse to row-major thread buffers (grown 0 times)
   +    0.000s (  0%) Transpose
   +    0.000s ( 27%) Waiting
   0.000s (  6%) Rereading 2 columns due to out-of-sample type exceptions
   0.001s        Total
Column 1 <<>> bumped from 'bool8' to 'float64' due to <<2023-10-12T06:53:53>> on row 0
Column 2 <<t>> bumped from 'bool8' to 'string' due to <<123Z>> on row 0
                    V1    t
1: 2023-10-12 06:53:53 123Z
Warning message:
In fread(text = "t\n2023-10-12T06:53:53,123Z", verbose = TRUE) :
  Detected 1 column names but the data has 2 columns (i.e. invalid file). Added 1 extra default column name for the first column which is guessed to be row names or an index. Use setnames() afterwards if this guess is not correct, or fix the file write command that created the file to create a valid file.

As seen from the warning, fread() assumes too strongly that sep="," and finds two fields, which conflicts with just one field in the header.

This might be a weird edge case related to only having one column, though -- to be investigated further.

Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.