DOI-USGS / DOI-USGS/loadflex

get clever about aggregating intervals via rloadest

Open
#224 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
16
Forks
18
PR merge metrics
No merged PRs in 30d

Description

rloadest has that limit of 176,000 rows for aggregation. many aggregation options should be small enough that even if the whole dataset is huge, a subset of the data would cover each entire aggregation interval and could be sent to rloadest (predLoad/predConc) separately from the rest of the data.

We currently do data chunking with chunks really close to 176,000. We could probably choose the chunk sizes better to make it so that more large datasets work fine. Code to modify is here: https://github.com/wdwatkins/loadflex/blob/9e4e520e0b88a04ce594aa1e7b5048e26dae0e2a/R/loadReg2.R#L356-L379

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.