Line numbers in error messages should refer to the raw file

Open
#2,428 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
r
Domain
data-engineering

Research direction

Start by reproducing the two fread examples from the issue and inspect the fread parsing and error-reporting path in data.table. Trace how quoted line breaks affect the reported line number, then verify that malformed records report their line in the raw file rather than the sampled or logical record position.

Written by the indexing model from the issue text.

Description

fread

Consider the following example:

src1 <- 'col1,col2
a,b
c,d,e'

fread(src1)
# Error in fread(src1) : 
#   Line 3 from sampling jump 0 starting <<c,d,e>> has more than the expected 2 fields. Separator ',' occurs at position 4 which is character 2 of the last field: <<d,e>>. Consider setting 'comment.char=' if there is a trailing comment to be ignored.

The error message reports a problem on line 3, which is fine.

Now suppose b contains a line break:

src2 <- 'col1,col2
a,"b1 
b2"
c,d,e'

fread(src2)
# Error in fread(src2) : 
#   Line 3 from sampling jump 0 starting <<c,d,e>> has more than the expected 2 fields. Separator ',' occurs at position 4 which is character 2 of the last field: <<d,e>>. Consider setting 'comment.char=' if there is a trailing comment to be ignored.

The problem is reported to be on line 3, which is wrong.

This could seem harmless, but when reading a very large CSV with lots of records containing line breaks, the line number in the message is totally wrong, and we end up with no clue to fix the misformatted file.

sessionInfo()

R version 3.4.2 (2017-09-28)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

Matrix products: default

locale:
[1] LC_COLLATE=French_France.1252  LC_CTYPE=French_France.1252    LC_MONETARY=French_France.1252 LC_NUMERIC=C                  
[5] LC_TIME=French_France.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.10.5

loaded via a namespace (and not attached):
[1] compiler_3.4.2 tools_3.4.2
Dominant language
R
Stars
3.9k
Forks
1.1k
Avg merge
14h 4m
Merged PRs (30d)
4

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Rdatatable/data.table

All issues in Rdatatable/data.table

Similar issues

More R issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.