DOI-USGS / DOI-USGS/streamMetabolizer
Errors: vector memory exhausted (limit reached?)
- Dominant language
- Stan
- Stars
- 47
- Forks
- 25
- PR merge metrics
- No merged PRs in 30d
Description
### Brief problem description
My dataset appears to be too large for streamMetabolizer to run on my computer. I am estimating metabolism from the Columbia River. Data set size is ~ 66,000 entries at 15 min intervals so about 690 days of metabolism, not too long. It runs roughly overnight (24,000 seconds) and returns the following error (error 1-3 are about tidyverse deprecations). It saves a small output file, but there are no mcmc samples in the file.
### What you saw on your computer
4: In metab_fun(specs = specs, data = data, data_daily = data_daily, :
Modeling failed
Warnings:
error in running command
There were 4 chains where the estimated Bayesian Fraction of Missing Information was low. See
http://mc-stan.org/misc/warnings.html#bfmi-low
Examine the pairs() plot to diagnose sampling problems Errors:
vector memory exhausted (limit reached?)
This last error suggests a memory problem. I am running a 2015 Macbook Pro with 16 GB memory. Run is on 4 cores. I am sampling 1000 warm up and 1000 saved steps. When I run a file with about 1/10 of the days and 500 warm up and saved steps, it works fine, and I get the expected results. That the small file works suggest that the data or model output is too big for my computer. Thus there is no problem per se, rather this is a PSA that 700 days of metabolism and 100 steps might be too much for my laptop. Time to run on a cluster.
Include all code you ran (a minimal example) and all console output, errors, and warnings. Include a data file if needed.
```r
load("CRall_sm.RData") #load data fram ready to go for sM, Bob can supply if needed
cr_name <- mm_name(type='bayes', pool_K600='normal', err_obs_iid = T, err_proc_iid =T)
cr_specs <- specs(cr_name, K600_daily_meanlog_meanlog=1, K600_daily_meanlog_sdlog=0.75, K600_daily_sdlog_sigma=0.1, burnin_steps=1000,
saved_steps=1000)
CRall_fit <- metab(cr_specs, data=CRall_sm, info=c(site='Columbia, date/time correction', source='Bob Hall'))
```
### Session information
Run the following code line (or `sessionInfo()` if that doesn't work) and paste in your output.
```r
> devtools::session_info()
─ Session info ────────────────────────────────────────────────────────────────────────────
setting value
version R version 4.0.3 (2020-10-10)
os macOS Mojave 10.14.6
system x86_64, darwin17.0
ui RStudio
language (EN)
collate en_US.UTF-8
ctype en_US.UTF-8
tz America/Denver
date 2021-01-21
─ Packages ────────────────────────────────────────────────────────────────────────────────
package * version date lib source
assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.0.2)
backports 1.1.10 2020-09-15 [1] CRAN (R 4.0.2)
base64enc 0.1-3 2015-07-28 [1] CRAN (R 4.0.2)
bayesplot 1.7.2 2020-05-28 [1] CRAN (R 4.0.2)
callr 3.5.1 2020-10-13 [1] CRAN (R 4.0.2)
cli 2.1.0 2020-10-12 [1] CRAN (R 4.0.2)
codetools 0.2-16 2018-12-24 [1] CRAN (R 4.0.3)
colorspace 1.4-1 2019-03-18 [1] CRAN (R 4.0.2)
colourpicker 1.1.0 2020-09-14 [1] CRAN (R 4.0.2)
crayon 1.3.4 2017-09-16 [1] CRAN (R 4.0.2)
crosstalk 1.1.0.1 2020-03-13 [1] CRAN (R 4.0.2)
curl 4.3 2019-12-02 [1] CRAN (R 4.0.1)
desc 1.2.0 2018-05-01 [1] CRAN (R 4.0.2)
deSolve 1.28 2020-03-08 [1] CRAN (R 4.0.2)
devtools 2.3.2 2020-09-18 [1] CRAN (R 4.0.2)
digest 0.6.26 2020-10-17 [1] CRAN (R 4.0.2)
dplyr * 1.0.2 2020-08-18 [1] CRAN (R 4.0.2)
DT 0.16 2020-10-13 [1] CRAN (R 4.0.2)
dygraphs 1.1.1.6 2018-07-11 [1] CRAN (R 4.0.2)
ellipsis 0.3.1 2020-05-15 [1] CRAN (R 4.0.2)
fansi 0.4.1 2020-01-08 [1] CRAN (R 4.0.2)
farver 2.0.3 2020-01-16 [1] CRAN (R 4.0.2)
fastmap 1.0.1 2019-10-08 [1] CRAN (R 4.0.2)
fs 1.5.0 2020-07-31 [1] CRAN (R 4.0.2)
generics 0.1.0 2020-10-31 [1] CRAN (R 4.0.2)
ggplot2 * 3.3.2 2020-06-19 [1] CRAN (R 4.0.2)
ggridges 0.5.2 2020-01-12 [1] CRAN (R 4.0.2)
glue 1.4.2 2020-08-27 [1] CRAN (R 4.0.2)
gridExtra 2.3 2017-09-09 [1] CRAN (R 4.0.2)
gtable 0.3.0 2019-03-25 [1] CRAN (R 4.0.2)
gtools 3.8.2 2020-03-31 [1] CRAN (R 4.0.2)
hms 0.5.3 2020-01-08 [1] CRAN (R 4.0.2)
htmltools 0.5.0 2020-06-16 [1] CRAN (R 4.0.2)
htmlwidgets 1.5.2 2020-10-03 [1] CRAN (R 4.0.2)
httpuv 1.5.4 2020-06-06 [1] CRAN (R 4.0.2)
igraph 1.2.6 2020-10-06 [1] CRAN (R 4.0.2)
inline 0.3.16 2020-09-06 [1] CRAN (R 4.0.2)
jsonlite 1.7.1 2020-09-07 [1] CRAN (R 4.0.2)
labeling 0.4.2 2020-10-20 [1] CRAN (R 4.0.3)
LakeMetabolizer 1.5.0 2016-06-23 [1] CRAN (R 4.0.2)
later 1.1.0.1 2020-06-05 [1] CRAN (R 4.0.2)
lattice 0.20-41 2020-04-02 [1] CRAN (R 4.0.3)
lazyeval 0.2.2 2019-03-15 [1] CRAN (R 4.0.2)
lifecycle 0.2.0 2020-03-06 [1] CRAN (R 4.0.2)
loo 2.3.1 2020-07-14 [1] CRAN (R 4.0.2)
lubridate * 1.7.9.2 2020-11-13 [1] CRAN (R 4.0.2)
magrittr 1.5 2014-11-22 [1] CRAN (R 4.0.2)
markdown 1.1 2019-08-07 [1] CRAN (R 4.0.2)
matrixStats 0.57.0 2020-09-25 [1] CRAN (R 4.0.2)
memoise 1.1.0 2017-04-21 [1] CRAN (R 4.0.2)
mime 0.9 2020-02-04 [1] CRAN (R 4.0.2)
miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.0.2)
munsell 0.5.0 2018-06-12 [1] CRAN (R 4.0.2)
pillar 1.4.6 2020-07-10 [1] CRAN (R 4.0.2)
pkgbuild 1.1.0 2020-07-13 [1] CRAN (R 4.0.2)
pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.0.2)
pkgload 1.1.0 2020-05-29 [1] CRAN (R 4.0.2)
plyr * 1.8.6 2020-03-03 [1] CRAN (R 4.0.2)
prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.0.2)
processx 3.4.4 2020-09-03 [1] CRAN (R 4.0.2)
promises 1.1.1 2020-06-09 [1] CRAN (R 4.0.2)
ps 1.4.0 2020-10-07 [1] CRAN (R 4.0.2)
purrr 0.3.4 2020-04-17 [1] CRAN (R 4.0.2)
R6 2.4.1 2019-11-12 [1] CRAN (R 4.0.2)
Rcpp 1.0.5 2020-07-06 [1] CRAN (R 4.0.2)
RcppParallel 5.0.2 2020-06-24 [1] CRAN (R 4.0.2)
readr * 1.4.0 2020-10-05 [1] CRAN (R 4.0.2)
remotes 2.2.0 2020-07-21 [1] CRAN (R 4.0.2)
reshape2 1.4.4 2020-04-09 [1] CRAN (R 4.0.2)
rLakeAnalyzer 1.11.4.1 2019-06-09 [1] CRAN (R 4.0.2)
rlang 0.4.8 2020-10-08 [1] CRAN (R 4.0.2)
rprojroot 1.3-2 2018-01-03 [1] CRAN (R 4.0.2)
rsconnect 0.8.16 2019-12-13 [1] CRAN (R 4.0.2)
rstan * 2.21.3 2020-10-20 [1] Github (stan-dev/rstan@ae36e6f)
rstudioapi 0.11 2020-02-07 [1] CRAN (R 4.0.2)
scales 1.1.1 2020-05-11 [1] CRAN (R 4.0.2)
sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 4.0.2)
shiny * 1.5.0 2020-06-23 [1] CRAN (R 4.0.2)
shinyjs 2.0.0 2020-09-09 [1] CRAN (R 4.0.2)
shinystan * 2.5.0 2018-05-01 [1] CRAN (R 4.0.2)
shinythemes 1.1.2 2018-11-06 [1] CRAN (R 4.0.2)
StanHeaders * 2.21.0-6 2020-08-16 [1] CRAN (R 4.0.2)
streamMetabolizer * 0.11.4 2020-10-22 [1] Github (USGS-R/streamMetabolizer@4a7ba9c)
stringi 1.5.3 2020-09-09 [1] CRAN (R 4.0.2)
stringr 1.4.0 2019-02-10 [1] CRAN (R 4.0.2)
testthat 2.3.2 2020-03-02 [1] CRAN (R 4.0.2)
threejs 0.3.3 2020-01-21 [1] CRAN (R 4.0.2)
tibble 3.0.4 2020-10-12 [1] CRAN (R 4.0.2)
tidyr * 1.1.2 2020-08-27 [1] CRAN (R 4.0.2)
tidyselect 1.1.0 2020-05-11 [1] CRAN (R 4.0.2)
unitted 0.2.9 2020-10-22 [1] Github (appling/unitted@d1f1172)
usethis 2.0.0 2020-12-10 [1] CRAN (R 4.0.2)
V8 3.2.0 2020-06-19 [1] CRAN (R 4.0.2)
vctrs 0.3.4 2020-08-29 [1] CRAN (R 4.0.2)
withr 2.3.0 2020-09-22 [1] CRAN (R 4.0.2)
xfun 0.18 2020-09-29 [1] CRAN (R 4.0.2)
xtable 1.8-4 2019-04-21 [1] CRAN (R 4.0.2)
xts 0.12.1 2020-09-09 [1] CRAN (R 4.0.2)
yaml 2.2.1 2020-02-01 [1] CRAN (R 4.0.2)
zoo 1.8-8 2020-05-02 [1] CRAN (R 4.0.2)
```
Contributor guide
Assessment
This issue has not been assessed yet.