Warn users if multicore summarise_draws risks exhausting available ram
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 171
- Forks
- 26
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 3
Description
My first big test of multicore summarise_draws on a really big (~9 GB in memory) array plowed straight into the swap on my 40 GB machine and I had to kill it. Second test, after reducing cores from 4 to 3, was awesome!
If a naive user runs their big model for days and then overenthusiastically runs summarise_draws prior to saving, they could be in a world of hurt.
Given that the main use case for multicore processing with summarise_draws involves really big draws arrays, we could consider adding a .ram_safety = TRUE argument, and then doing:
if (.cores > 1 & .ram_safety & (memuse::memuse(object.size(x)) > (memuse::Sys.meminfo()$totalram / (.cores + 1)))) {
warning(paste("The memory requirements of multi-core processing with", .cores, "cores may exceed the RAM on your system.",
"Consider using fewer cores. If you wish to proceed with", .cores, "cores, set `.ram_safety = FALSE`",
"and save any important objects prior to running."))
} else {
# Do the computation
}
Main drawback is the extra package requirement for multicore (which could go in suggests).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the multicore path of summarise_draws and inspect how package dependencies are declared, especially whether memuse can be added as a suggested dependency. Define the RAM warning and opt-out behavior described in the issue, then verify that multicore runs warn when estimated memory exceeds available RAM and proceed when safety is disabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100