out-of-memory warnings/solutions
- Dominant language
- R
- Stars
- 19
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
If the user asks for many GCMs/runs/periods/scenarios/years across many (many) locations, they can easily run out of memory to extract cliamte values and the large `data.table` of climate values to be downscaled.
I've seen this happen when asking for all GCMs/scenarios/periods x 3 runs for 2 700 000 point coordinates, using a 32Gb machine. The error is of the type `std::bac_alloc` which can easily be made more intuitive for the user with some messaging.
We could also foresee having `climr` actually _deal_ with this problem by, e.g.:
1. extracting/downscaling by subsets of points or combinations of climate model parameters
2. saving each subset to a csv file with `write.csv(..., append = TRUE)`
See https://stackoverflow.com/questions/78170318/error-stdbad-alloc-using-terraextract-on-large-stack-and-many-points -- extracting only the unique raster locations is not a good solution because 1) results are different and 2) the user will still run out of memory when expanding back to the full set of points.
Contributor guide
Assessment
This issue has not been assessed yet.