stan-dev / stan-dev/posterior

Warn users if multicore summarise_draws risks exhausting available ram

Open
#148 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation efficiency
Dominant language
R
Stars
171
Forks
26
Avg merge
2d 18h
Merged PRs (30d)
3

Description

My first big test of multicore summarise_draws on a really big (~9 GB in memory) array plowed straight into the swap on my 40 GB machine and I had to kill it. Second test, after reducing cores from 4 to 3, was awesome!

If a naive user runs their big model for days and then overenthusiastically runs summarise_draws prior to saving, they could be in a world of hurt.

Given that the main use case for multicore processing with summarise_draws involves really big draws arrays, we could consider adding a .ram_safety = TRUE argument, and then doing:

if (.cores > 1 & .ram_safety & (memuse::memuse(object.size(x)) > (memuse::Sys.meminfo()$totalram / (.cores + 1)))) {
  warning(paste("The memory requirements of multi-core processing with", .cores, "cores may exceed the RAM on your system.",
                "Consider using fewer cores. If you wish to proceed with", .cores, "cores, set `.ram_safety = FALSE`",
                "and save any important objects prior to running."))
} else {
  # Do the computation
}

Main drawback is the extra package requirement for multicore (which could go in suggests).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the multicore path of summarise_draws and inspect how package dependencies are declared, especially whether memuse can be added as a suggested dependency. Define the RAM warning and opt-out behavior described in the issue, then verify that multicore runs warn when estimated memory exceeds available RAM and proceed when safety is disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.