h2oai / h2oai/datatable

group by (datatable vs pandas)

Open
#2,551 12 comments 0 reactions 0 assignees View on GitHub
groupby performance
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

Hi,

I tested a `group_by` operation in a 9.6GB file, my results don't agree with [this benchmark](https://h2oai.github.io/db-benchmark/).

env:
2 x Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Ram 256G
```
Python 3.7.1 (default, Dec 14 2018, 19:28:38)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help.
```
Test: **Read**
```
import datatable as dt
%timeit dt.fread("/projects/salmonella/data/cgr.liv.ac.uk/parsed/cgr.parsed.features.tsv")
#42.3 s ± 3.05 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
import pandas as pd
%timeit pd.read_csv("/projects/salmonella/data/cgr.liv.ac.uk/parsed/cgr.parsed.features.tsv", sep = "\t")
#1min 28s ± 347 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

I also try figure out the conversion from `dataframe` to `pandas`:
```
%timeit dfp = df.to_pandas()
57.8 s ± 1.81 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
Too expensive conversions, there is others package that converts R data.frame to Pandas really fast.

But the point is, I tried the `group_by` with `count`:
test: **group by**
```
#datatable
%timeit df[ : , dt.count() , dt.by("nucleotide")]
6.7 s ± 939 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
#panda
%timeit dfp.groupby("nucleotide").size()
2.47 s ± 41.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Or there is a better way to perform count by group with `datatable`?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.