NVIDIA / NVIDIA/cudf

[BUG] read_csv fails to correctly handle misplaced quotes

Open
#2,398 5 comments 0 reactions 0 assignees View on GitHub
bug cuIO libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**
Often csv files have misplaced quotes and sometime there is a quotation mark as a part of one of the string fields. This should not be interpreted as a quotation mark indicating that a field has delimiters in it and therefore uses `"`.
**Steps/Code to reproduce bug**
```
import cudf
df = cudf.read_csv('quoting.csv')
df.to_pandas()
```
```
a b c d e
0 1 192.16.1.2 /abc/def/ghi 200 1
1 2 nvidia.com /abc/def/ghi.html 200 1
2 3 0.0.0.0 /images",500,1\n4,0.0.0.0,/abc/def,200,2\n5,ra... -1 -1
```
**Expected behavior**
```
import pandas as pd
df = pd..read_csv('quoting.csv')
df
```
```
a b c d e
0 1 192.16.1.2 /abc/def/ghi 200 1
1 2 nvidia.com /abc/def/ghi.html 200 1
2 3 0.0.0.0 /images" 500 1
3 4 0.0.0.0 /abc/def 200 2
4 5 rapids.ai /images 200 2
```

Workaround: Use cudf.read_csv with `quoting=3`. Pandas gives correct output for all quotation modes.

**Environment overview (please complete the following information)**
- Environment location: Docker nightly
- Method of cuDF install: Docker
- `docker pull rapidsai/rapidsai-nightly:0.9-cuda10.0-runtime-ubuntu16.04-gcc5-py3.7`

**Additional context**
I might be wrong but maybe checking if opening quote exists just after delimiter and the second one before another delimiter might be the way? (Just a guess)

[quoting.csv.zip](https://github.com/rapidsai/cudf/files/3433203/quoting.csv.zip)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.