h2oai / h2oai/datatable

Need an option to skip bad lines, like pandas' on_bad_line='skip'

Open
#3,087 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

I was dealing with some log files of about 78G size.
Even using fill=True, I got errors:
![image](https://user-images.githubusercontent.com/87395223/126255317-9a80f0ea-055f-45da-b844-58578bb0b302.png)

IOError: Too many fields on line 104485: expected 20 but more are present.

It's just because there's 1 bad line with one more column.
If I set columns=list(range(21)), fill=True
I got another error:
ValueError: Input contains 20 columns, whereas columns parameter specifies only 21 columns

These log files are generated everyday, so I can't edit it before throwing to datatable.

So, I hope an option like " error='skip' " or so.

Environment: Python 3.9.6 x64, datatable 1.0.0, vscode 1.58.2

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.