[BUG] CSV reader cannot handle unquoted quote character appearing in a field
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
This is directly from https://github.com/NVIDIA/spark-rapids/issues/6435 If you have a field like `abc""` in a CSV file the cudf CSV parser stops processing more data.
**Steps/Code to reproduce bug**
Create a file `test.csv` with the following data in it.
```
134324937434,#1991 N Grayhawk,"",Menlo Park,89025,AB,United States,US
208564744937,"63,trevion Way","",st Lothian,h7f4h8,"",United Kingdom,GB
132709376823,16 Oakland PARK RD,"",ring,l1w1e4,South,Canada,CA
224867848652,7 kingwell Court,"",United,s7jd9,South United,United Kingdom,GB
169636884295,30 cartuja Road,"",Halifax,L0R 9p2,ON,Canada,CA
859473321609,Street,"",Manchester,92220,OR,United States,US
141096112545,99 rue des,"",Australia,jsd9je,"",France,FR
160397658930,5 Rise,"",walligshngton,RY6 8LT,FORT,United Kingdom,GB
726367494002,1852 Townsend st,666,Wallsend,90382,CA,United States,US
187644735867,Bärbel-HAMPDEN-Ping 37,"",Miami,13355,"",Kingdom,ZZ
948475348324,155 sw City ct,Rochdale,Germany,30864,FL,Australia,QQ
164083193213,abc"","",Jerez Fra.,11401,Cadiz,Spain,ES
198732413077,3p Grove Rochdale road,BAW,Fulifax,HX4 trW,"",Israel,GB
227433927227,95 novem blvd,"",RAW VILLAGE,3173,XYZ,Australia,IL
```
Now try to read it using CUDF. The last two rows are skipped, and the `acb""` is read back missing the last `"`
(From spark using the rapids plugin for apache spark)
```
+------------+--------------------+--------+-------------+-------+------------+--------------+---+
| _c0| _c1| _c2| _c3| _c4| _c5| _c6|_c7|
+------------+--------------------+--------+-------------+-------+------------+--------------+---+
|134324937434| #1991 N Grayhawk| null| Menlo Park| 89025| AB| United States| US|
|208564744937| 63,trevion Way| null| st Lothian| h7f4h8| null|United Kingdom| GB|
|132709376823| 16 Oakland PARK RD| null| ring| l1w1e4| South| Canada| CA|
|224867848652| 7 kingwell Court| null| United| s7jd9|South United|United Kingdom| GB|
|169636884295| 30 cartuja Road| null| Halifax|L0R 9p2| ON| Canada| CA|
|859473321609| Street| null| Manchester| 92220| OR| United States| US|
|141096112545| 99 rue des| null| Australia| jsd9je| null| France| FR|
|160397658930| 5 Rise| null|walligshngton|RY6 8LT| FORT|United Kingdom| GB|
|726367494002| 1852 Townsend st| 666| Wallsend| 90382| CA| United States| US|
|187644735867|Bärbel-HAMPDEN-Pi...| null| Miami| 13355| null| Kingdom| ZZ|
|948475348324| 155 sw City ct|Rochdale| Germany| 30864| FL| Australia| QQ|
|164083193213| abc"| null| Jerez Fra.| 11401| Cadiz| Spain| ES|
+------------+--------------------+--------+-------------+-------+------------+--------------+---+
```
Without the plugin I get back
```
+------------+--------------------+--------+-------------+-------+------------+--------------+---+
| _c0| _c1| _c2| _c3| _c4| _c5| _c6|_c7|
+------------+--------------------+--------+-------------+-------+------------+--------------+---+
|134324937434| #1991 N Grayhawk| null| Menlo Park| 89025| AB| United States| US|
|208564744937| 63,trevion Way| null| st Lothian| h7f4h8| null|United Kingdom| GB|
|132709376823| 16 Oakland PARK RD| null| ring| l1w1e4| South| Canada| CA|
|224867848652| 7 kingwell Court| null| United| s7jd9|South United|United Kingdom| GB|
|169636884295| 30 cartuja Road| null| Halifax|L0R 9p2| ON| Canada| CA|
|859473321609| Street| null| Manchester| 92220| OR| United States| US|
|141096112545| 99 rue des| null| Australia| jsd9je| null| France| FR|
|160397658930| 5 Rise| null|walligshngton|RY6 8LT| FORT|United Kingdom| GB|
|726367494002| 1852 Townsend st| 666| Wallsend| 90382| CA| United States| US|
|187644735867|Bärbel-HAMPDEN-Pi...| null| Miami| 13355| null| Kingdom| ZZ|
|948475348324| 155 sw City ct|Rochdale| Germany| 30864| FL| Australia| QQ|
|164083193213| abc""| null| Jerez Fra.| 11401| Cadiz| Spain| ES|
|198732413077|3p Grove Rochdale...| BAW| Fulifax|HX4 trW| null| Israel| GB|
|227433927227| 95 novem blvd| null| RAW VILLAGE| 3173| XYZ| Australia| IL|
+------------+--------------------+--------+-------------+-------+------------+--------------+---+
```
Which is also what I get back from pandas.
```
>>> pd.read_csv("./test.csv", header=None)
0 1 2 3 4 5 6 7
0 134324937434 #1991 N Grayhawk NaN Menlo Park 89025 AB United States US
1 208564744937 63,trevion Way NaN st Lothian h7f4h8 NaN United Kingdom GB
2 132709376823 16 Oakland PARK RD NaN ring l1w1e4 South Canada CA
3 224867848652 7 kingwell Court NaN United s7jd9 South United United Kingdom GB
4 169636884295 30 cartuja Road NaN Halifax L0R 9p2 ON Canada CA
5 859473321609 Street NaN Manchester 92220 OR United States US
6 141096112545 99 rue des NaN Australia jsd9je NaN France FR
7 160397658930 5 Rise NaN walligshngton RY6 8LT FORT United Kingdom GB
8 726367494002 1852 Townsend st 666 Wallsend 90382 CA United States US
9 187644735867 Bärbel-HAMPDEN-Ping 37 NaN Miami 13355 NaN Kingdom ZZ
10 948475348324 155 sw City ct Rochdale Germany 30864 FL Australia QQ
11 164083193213 abc"" NaN Jerez Fra. 11401 Cadiz Spain ES
12 198732413077 3p Grove Rochdale road BAW Fulifax HX4 trW NaN Israel GB
13 227433927227 95 novem blvd NaN RAW VILLAGE 3173 XYZ Australia IL
```
**Expected behavior**
CUDF returns the same result as Pandas and Spark.
Contributor guide
Assessment
This issue has not been assessed yet.