stan-dev / stan-dev/rstan

Failure to dump large array with "stan_rdump"

Open
#740 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
1.1k
Forks
266
Avg merge
2h 56m
Merged PRs (30d)
1

Description

Summary:

Failure to allocate memory when dumping large array with function stan_rdump.

Description:

I tried to dump results extracted from a stanfit object into file using stan_rdump function. When I do so, I encountered an error:

Error in gsub(addnlpat, "\1\n", str) :
'Calloc' could not allocate memory (18446744071562067968 of 1 bytes)

I thought the problem is that the array is too large to dump (77112000 elements, 588.3Mb). Hope someone could rewrite the stan_rdump function to work with large objects.

e.g. something like this:

rdump <- function (list, file = "", envir=parent.frame())
{
  file <- file(file, "a")
  for (v in list) {
    vv <- get(v, envir)
    vvdim <- dim(vv)
    cat(v, " <- \n", file = file, sep = "")
    if (length(vv) == 0) {
      str <- paste0("structure(integer(0), ")
    }
    else {
      str <- paste0("structure(c(", paste(as.vector(vv),
                                          collapse = ", "), "),")
    }
    cat(str, "\n", ".Dim = c(", paste(vvdim, collapse = ", "), "))\n",
      file = file,
      sep = "")
  }
}
RStan Version:

2.19.2

R Version:

The version of R you are running (e.g., from R.version.string)
R version 3.6.1 (2019-07-05)

Operating System:

macOS 10.15.3

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the stan_rdump entry point and reproduce the failure with the reported 77,112,000-element array on the stated R and RStan versions. Trace the allocation involved in constructing and writing the dump; done means a large array can be dumped without the reported Calloc error.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.