h2oai / h2oai/datatable

Improve headers detection logic when all columns are of "string" type

Open
#946 0 comments 0 reactions 0 assignees View on GitHub
fread improve low priority
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

Consider a file like this:
```
A,B,C
D,E,F
```
Since all fields (and headers) are of type "string", we presume `header = true`, and print a verbose message that "no better guess is possible". However it **is** possible to have a better guess, by testing whether there are any repeating values within each column in the first 100 rows. For example, if the file is
```
a,b,c,d
a,h,i,j
a,e,c,
z,e,n,m
```
then it is more reasonable to assume that the first row is data, since it would be very uncommon for a column's header to have the same value as one of the cells.

An example file: https://s3.amazonaws.com/h2o-public-test-data/fread/Mushroom.gz

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.