rstan crashes R (and RStudio) when generated quantities contain array of large matrices

Summary:

rstan crashes R (and RStudio) when the generated quantities block contains an array of large matrices, for example, matrix[5000, 5] testobject[50].

Description:

Main problem: The R session crashes and takes down RStudio with it when the generated quantities block of a stan model contains an array of large matrices, for example matrix[5000, 5] testobject[50].

Additional notes: The model runs fine for arrays of very small matrices, for example, matrix[50, 5] testobject[50], in which case the rstan function call completes in about 2 seconds. The model does not crash for arrays of medium size matrices, for example, matrix[500, 5] testobject[50], but takes disproportionally longer to complete (about 380 seconds).

This crash is likely not caused by a lack of available memory, given that there were more than 64GB of free memory at the time of the crash and memory usage of the R session only went up to 0.7GB by the time the crash occurred.

Reproducible Steps:

library(Rcpp)

test_model = rstan::stan_model(
    model_code = "data {
      int nrow;
      int ncol;
      int nmatrices;
    }
    generated quantities {
      matrix[nrow, ncol] testobject[nmatrices];

      for (i in 1:nmatrices) {
        testobject[i] = rep_matrix(0.0 + i, nrow, ncol);
      }
    }"
)

# takes about 2 seconds
system.time({
test_result1 = rstan::sampling(
    object = test_model,
    algorithm = "Fixed_param",
    chains = 1,
    iter = 1,
    verbose = TRUE,
    data = list(nrow = 50, ncol = 5, nmatrices = 50)
)
})

# takes about 380 seconds
system.time({
test_result2 = rstan::sampling(
    object = test_model,
    algorithm = "Fixed_param",
    chains = 1,
    iter = 1,
    verbose = TRUE,
    data = list(nrow = 500, ncol = 5, nmatrices = 50)
)
})

# crash
system.time({
test_result3 = rstan::sampling(
    object = test_model,
    algorithm = "Fixed_param",
    chains = 1,
    iter = 1,
    verbose = TRUE,
    data = list(nrow = 5000, ncol = 5, nmatrices = 50)
)
})

Current Output:

Crashed R/RStudio session

Expected Output:

No crash for arrays of large matrices; shorter runtime for array of medium size matrices. Any help is appreciated!

RStan Version:

2.16.2

R Version:

R version 3.4.2 (2017-09-28)

Operating System:

Windows Server 2016

Start by running the provided R script through rstan::stan_model and rstan::sampling, comparing the small, medium, and large matrix cases. Trace where generated quantities containing arrays of matrices are processed, then verify that the large case no longer crashes and that the medium case completes substantially faster.