h2oai / h2oai/datatable

performance regression when groupby acts on multiple columns

Open
#2,541 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

For example, we have a frame which the shape of it is(34284584, 18),
when by only acts on one column or a few columns, like:

`df[:, {"A":min(f.A), by("B")}] or df[:, {"A":min(f.A), by(["B", "C"])}]`

It takes about seconds.

But when by acts on multiple columns, the time-consuming increases to the minute level, like:

`df[:, {"A":min(f.A), by(["B", "C", "D", "E", "F", "H", "G", "J", "K", "M")}]`

Performance is even weaker than single-threaded pandas.

Is there a performance problem with groupby based on multiple columns?
Thanks a lot!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.