group by (datatable vs pandas)
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 164
- Avg merge
- 7h 31m
- Merged PRs (30d)
- 1
Description
Hi,
I tested a `group_by` operation in a 9.6GB file, my results don't agree with [this benchmark](https://h2oai.github.io/db-benchmark/).
env:
2 x Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz, Ram 256G
```
Python 3.7.1 (default, Dec 14 2018, 19:28:38)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help.
```
Test: **Read**
```
import datatable as dt
%timeit dt.fread("/projects/salmonella/data/cgr.liv.ac.uk/parsed/cgr.parsed.features.tsv")
#42.3 s ± 3.05 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
import pandas as pd
%timeit pd.read_csv("/projects/salmonella/data/cgr.liv.ac.uk/parsed/cgr.parsed.features.tsv", sep = "\t")
#1min 28s ± 347 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
I also try figure out the conversion from `dataframe` to `pandas`:
```
%timeit dfp = df.to_pandas()
57.8 s ± 1.81 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
Too expensive conversions, there is others package that converts R data.frame to Pandas really fast.
But the point is, I tried the `group_by` with `count`:
test: **group by**
```
#datatable
%timeit df[ : , dt.count() , dt.by("nucleotide")]
6.7 s ± 939 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
#panda
%timeit dfp.groupby("nucleotide").size()
2.47 s ± 41.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
Or there is a better way to perform count by group with `datatable`?
Contributor guide
Assessment
This issue has not been assessed yet.