DOI-USGS / DOI-USGS/streamMetabolizer

Errors: vector memory exhausted (limit reached?)

Open
#404 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Stan
Stars
47
Forks
25
PR merge metrics
No merged PRs in 30d

Description

### Brief problem description
My dataset appears to be too large for streamMetabolizer to run on my computer. I am estimating metabolism from the Columbia River. Data set size is ~ 66,000 entries at 15 min intervals so about 690 days of metabolism, not too long. It runs roughly overnight (24,000 seconds) and returns the following error (error 1-3 are about tidyverse deprecations). It saves a small output file, but there are no mcmc samples in the file.

### What you saw on your computer
4: In metab_fun(specs = specs, data = data, data_daily = data_daily, :
Modeling failed
Warnings:
error in running command
There were 4 chains where the estimated Bayesian Fraction of Missing Information was low. See
http://mc-stan.org/misc/warnings.html#bfmi-low
Examine the pairs() plot to diagnose sampling problems Errors:
vector memory exhausted (limit reached?)

This last error suggests a memory problem. I am running a 2015 Macbook Pro with 16 GB memory. Run is on 4 cores. I am sampling 1000 warm up and 1000 saved steps. When I run a file with about 1/10 of the days and 500 warm up and saved steps, it works fine, and I get the expected results. That the small file works suggest that the data or model output is too big for my computer. Thus there is no problem per se, rather this is a PSA that 700 days of metabolism and 100 steps might be too much for my laptop. Time to run on a cluster.

Include all code you ran (a minimal example) and all console output, errors, and warnings. Include a data file if needed.

```r
load("CRall_sm.RData") #load data fram ready to go for sM, Bob can supply if needed

cr_name <- mm_name(type='bayes', pool_K600='normal', err_obs_iid = T, err_proc_iid =T)
cr_specs <- specs(cr_name, K600_daily_meanlog_meanlog=1, K600_daily_meanlog_sdlog=0.75, K600_daily_sdlog_sigma=0.1, burnin_steps=1000,
saved_steps=1000)
CRall_fit <- metab(cr_specs, data=CRall_sm, info=c(site='Columbia, date/time correction', source='Bob Hall'))

```

### Session information

Run the following code line (or `sessionInfo()` if that doesn't work) and paste in your output.

```r
> devtools::session_info()

─ Session info ────────────────────────────────────────────────────────────────────────────
setting value
version R version 4.0.3 (2020-10-10)
os macOS Mojave 10.14.6
system x86_64, darwin17.0
ui RStudio
language (EN)
collate en_US.UTF-8
ctype en_US.UTF-8
tz America/Denver
date 2021-01-21

─ Packages ────────────────────────────────────────────────────────────────────────────────
package * version date lib source
assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.0.2)
backports 1.1.10 2020-09-15 [1] CRAN (R 4.0.2)
base64enc 0.1-3 2015-07-28 [1] CRAN (R 4.0.2)
bayesplot 1.7.2 2020-05-28 [1] CRAN (R 4.0.2)
callr 3.5.1 2020-10-13 [1] CRAN (R 4.0.2)
cli 2.1.0 2020-10-12 [1] CRAN (R 4.0.2)
codetools 0.2-16 2018-12-24 [1] CRAN (R 4.0.3)
colorspace 1.4-1 2019-03-18 [1] CRAN (R 4.0.2)
colourpicker 1.1.0 2020-09-14 [1] CRAN (R 4.0.2)
crayon 1.3.4 2017-09-16 [1] CRAN (R 4.0.2)
crosstalk 1.1.0.1 2020-03-13 [1] CRAN (R 4.0.2)
curl 4.3 2019-12-02 [1] CRAN (R 4.0.1)
desc 1.2.0 2018-05-01 [1] CRAN (R 4.0.2)
deSolve 1.28 2020-03-08 [1] CRAN (R 4.0.2)
devtools 2.3.2 2020-09-18 [1] CRAN (R 4.0.2)
digest 0.6.26 2020-10-17 [1] CRAN (R 4.0.2)
dplyr * 1.0.2 2020-08-18 [1] CRAN (R 4.0.2)
DT 0.16 2020-10-13 [1] CRAN (R 4.0.2)
dygraphs 1.1.1.6 2018-07-11 [1] CRAN (R 4.0.2)
ellipsis 0.3.1 2020-05-15 [1] CRAN (R 4.0.2)
fansi 0.4.1 2020-01-08 [1] CRAN (R 4.0.2)
farver 2.0.3 2020-01-16 [1] CRAN (R 4.0.2)
fastmap 1.0.1 2019-10-08 [1] CRAN (R 4.0.2)
fs 1.5.0 2020-07-31 [1] CRAN (R 4.0.2)
generics 0.1.0 2020-10-31 [1] CRAN (R 4.0.2)
ggplot2 * 3.3.2 2020-06-19 [1] CRAN (R 4.0.2)
ggridges 0.5.2 2020-01-12 [1] CRAN (R 4.0.2)
glue 1.4.2 2020-08-27 [1] CRAN (R 4.0.2)
gridExtra 2.3 2017-09-09 [1] CRAN (R 4.0.2)
gtable 0.3.0 2019-03-25 [1] CRAN (R 4.0.2)
gtools 3.8.2 2020-03-31 [1] CRAN (R 4.0.2)
hms 0.5.3 2020-01-08 [1] CRAN (R 4.0.2)
htmltools 0.5.0 2020-06-16 [1] CRAN (R 4.0.2)
htmlwidgets 1.5.2 2020-10-03 [1] CRAN (R 4.0.2)
httpuv 1.5.4 2020-06-06 [1] CRAN (R 4.0.2)
igraph 1.2.6 2020-10-06 [1] CRAN (R 4.0.2)
inline 0.3.16 2020-09-06 [1] CRAN (R 4.0.2)
jsonlite 1.7.1 2020-09-07 [1] CRAN (R 4.0.2)
labeling 0.4.2 2020-10-20 [1] CRAN (R 4.0.3)
LakeMetabolizer 1.5.0 2016-06-23 [1] CRAN (R 4.0.2)
later 1.1.0.1 2020-06-05 [1] CRAN (R 4.0.2)
lattice 0.20-41 2020-04-02 [1] CRAN (R 4.0.3)
lazyeval 0.2.2 2019-03-15 [1] CRAN (R 4.0.2)
lifecycle 0.2.0 2020-03-06 [1] CRAN (R 4.0.2)
loo 2.3.1 2020-07-14 [1] CRAN (R 4.0.2)
lubridate * 1.7.9.2 2020-11-13 [1] CRAN (R 4.0.2)
magrittr 1.5 2014-11-22 [1] CRAN (R 4.0.2)
markdown 1.1 2019-08-07 [1] CRAN (R 4.0.2)
matrixStats 0.57.0 2020-09-25 [1] CRAN (R 4.0.2)
memoise 1.1.0 2017-04-21 [1] CRAN (R 4.0.2)
mime 0.9 2020-02-04 [1] CRAN (R 4.0.2)
miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.0.2)
munsell 0.5.0 2018-06-12 [1] CRAN (R 4.0.2)
pillar 1.4.6 2020-07-10 [1] CRAN (R 4.0.2)
pkgbuild 1.1.0 2020-07-13 [1] CRAN (R 4.0.2)
pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.0.2)
pkgload 1.1.0 2020-05-29 [1] CRAN (R 4.0.2)
plyr * 1.8.6 2020-03-03 [1] CRAN (R 4.0.2)
prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.0.2)
processx 3.4.4 2020-09-03 [1] CRAN (R 4.0.2)
promises 1.1.1 2020-06-09 [1] CRAN (R 4.0.2)
ps 1.4.0 2020-10-07 [1] CRAN (R 4.0.2)
purrr 0.3.4 2020-04-17 [1] CRAN (R 4.0.2)
R6 2.4.1 2019-11-12 [1] CRAN (R 4.0.2)
Rcpp 1.0.5 2020-07-06 [1] CRAN (R 4.0.2)
RcppParallel 5.0.2 2020-06-24 [1] CRAN (R 4.0.2)
readr * 1.4.0 2020-10-05 [1] CRAN (R 4.0.2)
remotes 2.2.0 2020-07-21 [1] CRAN (R 4.0.2)
reshape2 1.4.4 2020-04-09 [1] CRAN (R 4.0.2)
rLakeAnalyzer 1.11.4.1 2019-06-09 [1] CRAN (R 4.0.2)
rlang 0.4.8 2020-10-08 [1] CRAN (R 4.0.2)
rprojroot 1.3-2 2018-01-03 [1] CRAN (R 4.0.2)
rsconnect 0.8.16 2019-12-13 [1] CRAN (R 4.0.2)
rstan * 2.21.3 2020-10-20 [1] Github (stan-dev/rstan@ae36e6f)
rstudioapi 0.11 2020-02-07 [1] CRAN (R 4.0.2)
scales 1.1.1 2020-05-11 [1] CRAN (R 4.0.2)
sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 4.0.2)
shiny * 1.5.0 2020-06-23 [1] CRAN (R 4.0.2)
shinyjs 2.0.0 2020-09-09 [1] CRAN (R 4.0.2)
shinystan * 2.5.0 2018-05-01 [1] CRAN (R 4.0.2)
shinythemes 1.1.2 2018-11-06 [1] CRAN (R 4.0.2)
StanHeaders * 2.21.0-6 2020-08-16 [1] CRAN (R 4.0.2)
streamMetabolizer * 0.11.4 2020-10-22 [1] Github (USGS-R/streamMetabolizer@4a7ba9c)
stringi 1.5.3 2020-09-09 [1] CRAN (R 4.0.2)
stringr 1.4.0 2019-02-10 [1] CRAN (R 4.0.2)
testthat 2.3.2 2020-03-02 [1] CRAN (R 4.0.2)
threejs 0.3.3 2020-01-21 [1] CRAN (R 4.0.2)
tibble 3.0.4 2020-10-12 [1] CRAN (R 4.0.2)
tidyr * 1.1.2 2020-08-27 [1] CRAN (R 4.0.2)
tidyselect 1.1.0 2020-05-11 [1] CRAN (R 4.0.2)
unitted 0.2.9 2020-10-22 [1] Github (appling/unitted@d1f1172)
usethis 2.0.0 2020-12-10 [1] CRAN (R 4.0.2)
V8 3.2.0 2020-06-19 [1] CRAN (R 4.0.2)
vctrs 0.3.4 2020-08-29 [1] CRAN (R 4.0.2)
withr 2.3.0 2020-09-22 [1] CRAN (R 4.0.2)
xfun 0.18 2020-09-29 [1] CRAN (R 4.0.2)
xtable 1.8-4 2019-04-21 [1] CRAN (R 4.0.2)
xts 0.12.1 2020-09-09 [1] CRAN (R 4.0.2)
yaml 2.2.1 2020-02-01 [1] CRAN (R 4.0.2)
zoo 1.8-8 2020-05-02 [1] CRAN (R 4.0.2)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.