h2oai / h2oai/datatable

Improve reducers parallelization logic

Open
#3,081 0 comments 1 reaction 0 assignees View on GitHub
improve performance
Dominant language
C++
Stars
1.9k
Forks
164
Avg merge
7h 31m
Merged PRs (30d)
1

Description

Right now datatable reducers are only parallelized in terms of the groups. When there is one group, even if it is huge, datatable does all the work in single thread. For instance,

```python
from datatable import dt, f
DT = dt.Frame(range(10**9))
print(DT[:, dt.sum(f[0])])
```

will utilize one thread only and take significant amount of time to complete. We should improve the parallelization logic by taking into account not just the number of groups, but also their sizes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.