h2oai / h2oai/datatable

In fread skip_blank_lines default behavior is too strict

Open
#838 0 comments 0 reactions 0 assignees View on GitHub
fread improve low priority
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

Stemming from the discussion in #837, we want fread to be more lenient towards empty lines in the input. Currently, `skip_blank_lines` is `False` by default (raises an exception when a blank line is encountered). When set to `True`, the option will cause all empty lines to be skipped.

Ideally, the default behavior should be somewhere in between these two: if an empty line is found then skip it, provided the next line has the "correct" number of columns. Empty lines should not be skipped by default in single-column mode (instead they are treated as NAs or blanks).

In addition, current error message when encountering blank lines is bad: `RuntimeError: Expecting 4 cols but row 0 contains only 0 cols (sep=','). Consider fill=true. ""`. We want to be more explicit, for example: `FreadError: Empty line found on line N in the input, and text exists afterwards. Use skip_blank_lines=True or fill=True to ignore this error.`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.