[BUG] read_csv fails to correctly handle misplaced quotes
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
Often csv files have misplaced quotes and sometime there is a quotation mark as a part of one of the string fields. This should not be interpreted as a quotation mark indicating that a field has delimiters in it and therefore uses `"`.
**Steps/Code to reproduce bug**
```
import cudf
df = cudf.read_csv('quoting.csv')
df.to_pandas()
```
```
a b c d e
0 1 192.16.1.2 /abc/def/ghi 200 1
1 2 nvidia.com /abc/def/ghi.html 200 1
2 3 0.0.0.0 /images",500,1\n4,0.0.0.0,/abc/def,200,2\n5,ra... -1 -1
```
**Expected behavior**
```
import pandas as pd
df = pd..read_csv('quoting.csv')
df
```
```
a b c d e
0 1 192.16.1.2 /abc/def/ghi 200 1
1 2 nvidia.com /abc/def/ghi.html 200 1
2 3 0.0.0.0 /images" 500 1
3 4 0.0.0.0 /abc/def 200 2
4 5 rapids.ai /images 200 2
```
Workaround: Use cudf.read_csv with `quoting=3`. Pandas gives correct output for all quotation modes.
**Environment overview (please complete the following information)**
- Environment location: Docker nightly
- Method of cuDF install: Docker
- `docker pull rapidsai/rapidsai-nightly:0.9-cuda10.0-runtime-ubuntu16.04-gcc5-py3.7`
**Additional context**
I might be wrong but maybe checking if opening quote exists just after delimiter and the second one before another delimiter might be the way? (Just a guess)
[quoting.csv.zip](https://github.com/rapidsai/cudf/files/3433203/quoting.csv.zip)
Contributor guide
Assessment
This issue has not been assessed yet.