fread fails for properly quoted \n in file with \r as EOL

Start by reproducing the minimal example with fread() and the linked dt-test.csv input, then compare its parsing with data.table 1.10.4-3. Trace the fread() handling of carriage returns, quoted fields, and row boundaries. Done means the ^M separators create rows while carriage returns inside quoted cells do not, matching the expected four-row output.

I described this on stackoverflow, Hugh suggested it might be a valid issue to report here (https://stackoverflow.com/questions/50918972/data-table-fread-does-not-see-m-as-new-line-any-more)

This is a new problem with data.table 1.11.4, did not occur in 1.10.4-3.

Data sets distributed from Qualtrics sometimes have both "^M" to mark new lines and also there are "naked" carriage returns in the middle of data cells. This confuses R read.csv horribly, we changed over to fread and it has worked in past. Now it is not working. In all of the cases I have tested, the problem is in row 2 of the data frame where the questions are repeated and when the Qualtrics user includes a carriage return in their cut/paste, then that return is preserved in the data. I made an MRE data set, http://pj.freefaculty.org/scraps/dt-test.csv

Another element in this mess is the new-ish Excel style of quotes for character variables, which do not include quotation marks unless absolutely necessary. We see some cell entries with quotes and others without in the row 2 information.

I do not want to preserve the carriage returns that are in the middle of the cells. I want the "^M" to be seen as new lines.

# Minimal reproducible example

library(data.table)
fread("http://pj.freefaculty.org/scraps/dt-test.csv")

The output is as follows.
Note that the things that show as "^M" in Emacs are seen as "\r" in fread error message:

   id,y,x1,x2\rRespondentidentification,      "An
1:                            variable", "Working
2:                               input",    "more
                                  outcome
1:                                     on
2: input"\rResp1,33,22,33\rResp2,31,19,32
Warning message:
In fread("http://pj.freefaculty.org/scraps/dt-test.csv") :
  Found and resolved improper quoting in first 100 rows. If the fields are not quoted (e.g. field separator does not appear within any field), try quote="" to avoid this warning.

Desired result, which happened in previous version of data.table, is a new data object, with the CSV rows separated by the "^M" thing and ignoring the carriage returns in the middle of the input data for row 2.

id,y,x1,x2
Respondentidentification, "An outcome variable", "Working on input", "more input"
Resp1,33,22,33
Resp2,31,19,32

I expect these will be all character because of the questions in row 2.

I am not allowed to publish the real Qualtrics data set in this forum, but if one of the data.table developers is willing to write me at pauljohn@ku.edu, I'll email a copy if you will agree not to redistribute. But I do believe the MRE is representative.

# Output of sessionInfo()

R version 3.4.4 (2018-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] data.table_1.11.4

loaded via a namespace (and not attached):
[1] compiler_3.4.4