Inconsistent CSV dialect
Open
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 755
- Forks
- 128
- PR merge metrics
- No merged PRs in 30d
Description
If you look at the CSV file of this dataset you can see that it has inconsistent CSV dialect:
- column names are quoted with
" - attribute values are quoted with
'and'is escaped as\'
Quote:
"history","session","user"
0000,'whoami pwd ls dir vi source <1> source <1> exit',USER0
0001,'whereis <1> mkdir <1> vi <1> vi <1> ls source <1>',USER0
There is no way to configure Pandas.read_csv to read this file correctly. You can use
quotechar='\'',
doublequote=False,
escapechar='\\',`
but then you get " in column names. Also, it is unclear how is " escaped in column names (as "" or as \").
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the problem with the linked CSV file and the shown pandas.read_csv options. No repository file or test is named; done would require determining how this inconsistent dialect should be handled or documenting that the source file cannot be parsed unambiguously.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100