bcgov / bcgov/climr

out-of-memory warnings/solutions

Open
#237 7 comments 0 reactions 2 assignees Claimed by @kdaust View on GitHub
v1.0.0
Dominant language
R
Stars
19
Forks
6
PR merge metrics
No merged PRs in 30d

Description

If the user asks for many GCMs/runs/periods/scenarios/years across many (many) locations, they can easily run out of memory to extract cliamte values and the large `data.table` of climate values to be downscaled.

I've seen this happen when asking for all GCMs/scenarios/periods x 3 runs for 2 700 000 point coordinates, using a 32Gb machine. The error is of the type `std::bac_alloc` which can easily be made more intuitive for the user with some messaging.

We could also foresee having `climr` actually _deal_ with this problem by, e.g.:
1. extracting/downscaling by subsets of points or combinations of climate model parameters
2. saving each subset to a csv file with `write.csv(..., append = TRUE)`

See https://stackoverflow.com/questions/78170318/error-stdbad-alloc-using-terraextract-on-large-stack-and-many-points -- extracting only the unique raster locations is not a good solution because 1) results are different and 2) the user will still run out of memory when expanding back to the full set of points.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.