bcgov / bcgov/bcEpiRate

explore parallelization in operations using `group_by()`

Open
#21 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
8
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Our team has encountered a performance issue when using package functions that use `dplyr::group_by()` followed by `dplyr::summarise()` on very large dataframes. A common example is `get_ds_rt()`.

This seems to be a `dplyr` issue that is common across many usages.

A workaround is to split up the dataframe into chunks and run one chunk at a time. This could be parallelized by the user as well.

However, perhaps there is a way to do this within the package? It could be off by default and could be implemented by adding an argument in these functions such as `nthreads` where a user can specify how many threads they would be willing to use on the operation.

(e..g https://multidplyr.tidyverse.org/ ? or other similar packages)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.