h2oai / h2oai/datatable

fread string with NAs generates extra distinct group

Open
#2,808 0 comments 0 reactions 0 assignees View on GitHub
bug fread
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

```sh
wget https://raw.githubusercontent.com/h2oai/db-benchmark/cf255c174647ac437aa7a85751f6e65732a3cb9a/_data/groupby-datagen.R
Rscript groupby-datagen.R 1e9 1e2 5 0

## activate your pydt env
source ~/git/db-benchmark/pydatatable/py-pydatatable/bin/activate
python

import datatable as dt
from datatable import f, count
x = dt.fread('G1_1e9_1e2_5_0.csv', na_strings=[''])
print(x.nrows, flush=True)
#1000000000
x[f.id1=="", count()]
# | count
#-- + -------
# 0 | 2501132
#
#[1 row x 1 column]
x[isna(f.id1), count()]
# | count
#-- + --------
# 0 | 47505964
#
#[1 row x 1 column]
```

```sh
grep "^," G1_1e9_1e2_5_0.csv | wc -l
#50007096
```

**Note that this issue only manifests on 1e9 data size and not on smaller sizes**. Smaller sizes have exactly same number of distinct groups for this column.

- What was the expected behavior?

Properly read NAs.

- Your environment?

pydt 9bc7d05db2e35a480fb9aea7b570c1005776ae4d
python 3.6.7
ubuntu 16.04

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.