stan-dev / stan-dev/rstan

Random reproducibility issue

Open
#989 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
1.1k
Forks
266
Avg merge
2h 56m
Merged PRs (30d)
1

Description

Summary:

There seems to be a random reproducibility issue for some models.

Description:

While working on projpred's unit tests (PR stan-dev/projpred#277, but that's not important here), I realized that there is a reproducibility issue when running loo::kfold() for GAMMs originally fit by brms with the rstan backend. A main obstacle for debugging this is that the issue occurs randomly, i.e., sometimes results are reproducible and sometimes not. I would estimate it to occur about 50% of the time. Of course, due its random nature, debugging this issue will be hard. But I decided to put this on the issue tracker even if it might not be possible to solve this, in particular to notify others.

I'm not sure if this is really related to rstan or if it's rather related to brms, loo, or testthat (or a combination of these). The reasons why my first guess is rstan are:

  1. I did not encounter this issue with the cmdstanr backend in brms.
  2. I compared the stanfit objects from two differing runs and they were basically the same (in particular with regard to the seed argument), except for the draws (component <stanfit_object>@sim$samples). So I would guess brms calls rstan always with the same arguments and it's rstan which somewhere has this reproducibility issue.

So the fact that I'm referring to brms above and in the reprex below should not be understood as brms being necessary for this issue, but rather as brms being the first case where I encountered this.

Reproducible Steps:

As mentioned above, the issue is not reproducible deterministically, but it occurs randomly about 50% of the time. I have created a minimal R package where I was able to observe the issue when running R CMD check repeatedly. It might be possible to reduce that minimal package to a few lines of standalone R code, but I'm currently lacking the time to do so.

Current Output:

Not applicable.

Expected Output:

Not applicable.

RStan Version:

2.26.6 (from https://mc-stan.org/r-packages/, just like StanHeaders)

R Version:

R version 4.1.2 (2021-11-01)

Operating System:

Ubuntu 20.04.3 LTS

Complete session info:

Some more session info, in particular including brms and testthat in case they are needed:

library(brms)
library(testthat)
sessioninfo::session_info()

gives:

─ Session info ───────────────────────────────────────────
 setting  value
 version  R version 4.1.2 (2021-11-01)
 os       Ubuntu 20.04.3 LTS
 system   x86_64, linux-gnu
 ui       RStudio
 language (EN)
 collate  en_US.UTF-8
 ctype    en_US.UTF-8
 tz       Europe/Berlin
 date     2022-02-20
 rstudio  2021.09.2+382 Ghost Orchid (desktop)
 pandoc   2.5 @ /usr/bin/pandoc

─ Packages ───────────────────────────────────────────────
 package        * version    date (UTC) lib source
 abind            1.4-5      2016-07-21 [1] CRAN (R 4.0.4)
 assertthat       0.2.1      2019-03-21 [1] CRAN (R 4.0.4)
 backports        1.4.1      2021-12-13 [1] CRAN (R 4.1.2)
 base64enc        0.1-3      2015-07-28 [1] CRAN (R 4.0.4)
 bayesplot        1.8.1      2021-06-14 [1] CRAN (R 4.1.0)
 bridgesampling   1.1-2      2021-04-16 [1] CRAN (R 4.0.5)
 brio             1.1.3      2021-11-30 [1] CRAN (R 4.1.2)
 brms           * 2.16.8     2022-02-20 [1] Github (paul-buerkner/brms@39f6169)
 Brobdingnag      1.2-7      2022-02-03 [1] CRAN (R 4.1.2)
 callr            3.7.0      2021-04-20 [1] CRAN (R 4.0.5)
 checkmate        2.0.0      2020-02-06 [1] CRAN (R 4.0.4)
 cli              3.2.0      2022-02-14 [1] CRAN (R 4.1.2)
 coda             0.19-4     2020-09-30 [1] CRAN (R 4.0.4)
 codetools        0.2-18     2020-11-04 [4] CRAN (R 4.0.3)
 colorspace       2.0-2      2021-06-24 [1] CRAN (R 4.1.0)
 colourpicker     1.1.1      2021-10-04 [1] CRAN (R 4.1.1)
 crayon           1.5.0      2022-02-14 [1] CRAN (R 4.1.2)
 crosstalk        1.2.0      2021-11-04 [1] CRAN (R 4.1.2)
 curl             4.3.2      2021-06-23 [1] CRAN (R 4.1.0)
 DBI              1.1.2      2021-12-20 [1] CRAN (R 4.1.2)
 digest           0.6.29     2021-12-01 [1] CRAN (R 4.1.2)
 distributional   0.3.0      2022-01-05 [1] CRAN (R 4.1.2)
 dplyr            1.0.8      2022-02-08 [1] CRAN (R 4.1.2)
 DT               0.20       2021-11-15 [1] CRAN (R 4.1.2)
 dygraphs         1.1.1.6    2018-07-11 [1] CRAN (R 4.0.4)
 ellipsis         0.3.2      2021-04-29 [1] CRAN (R 4.0.5)
 emmeans          1.7.2      2022-01-04 [1] CRAN (R 4.1.2)
 estimability     1.3        2018-02-11 [1] CRAN (R 4.0.4)
 fansi            1.0.2      2022-01-14 [1] CRAN (R 4.1.2)
 farver           2.1.0      2021-02-28 [1] CRAN (R 4.0.4)
 fastmap          1.1.0      2021-01-25 [1] CRAN (R 4.0.4)
 generics         0.1.2      2022-01-31 [1] CRAN (R 4.1.2)
 ggplot2          3.3.5      2021-06-25 [1] CRAN (R 4.1.0)
 ggridges         0.5.3      2021-01-08 [1] CRAN (R 4.0.4)
 glue             1.6.1      2022-01-22 [1] CRAN (R 4.1.2)
 gridExtra        2.3        2017-09-09 [1] CRAN (R 4.0.4)
 gtable           0.3.0      2019-03-25 [1] CRAN (R 4.0.4)
 gtools           3.9.2      2021-06-06 [1] CRAN (R 4.1.0)
 htmltools        0.5.2      2021-08-25 [1] CRAN (R 4.1.1)
 htmlwidgets      1.5.4      2021-09-08 [1] CRAN (R 4.1.1)
 httpuv           1.6.5      2022-01-05 [1] CRAN (R 4.1.2)
 igraph           1.2.11     2022-01-04 [1] CRAN (R 4.1.2)
 inline           0.3.19     2021-05-31 [1] CRAN (R 4.1.0)
 jsonlite         1.7.3      2022-01-17 [1] CRAN (R 4.1.2)
 later            1.3.0      2021-08-18 [1] CRAN (R 4.1.1)
 lattice          0.20-45    2021-09-22 [4] CRAN (R 4.1.1)
 lifecycle        1.0.1      2021-09-24 [1] CRAN (R 4.1.1)
 loo              2.4.1.9000 2022-02-09 [1] Github (stan-dev/loo@4c85352)
 magrittr         2.0.2      2022-01-26 [1] CRAN (R 4.1.2)
 markdown         1.1        2019-08-07 [1] CRAN (R 4.0.4)
 MASS             7.3-55     2022-01-13 [4] CRAN (R 4.1.2)
 Matrix           1.4-0      2021-12-08 [4] CRAN (R 4.1.2)
 matrixStats      0.61.0     2021-09-17 [1] CRAN (R 4.1.1)
 mime             0.12       2021-09-28 [1] CRAN (R 4.1.1)
 miniUI           0.1.1.1    2018-05-18 [1] CRAN (R 4.0.4)
 multcomp         1.4-18     2022-01-04 [1] CRAN (R 4.1.2)
 munsell          0.5.0      2018-06-12 [1] CRAN (R 4.0.4)
 mvtnorm          1.1-3      2021-10-08 [1] CRAN (R 4.1.1)
 nlme             3.1-155    2022-01-13 [4] CRAN (R 4.1.2)
 pillar           1.7.0      2022-02-01 [1] CRAN (R 4.1.2)
 pkgbuild         1.3.1      2021-12-20 [1] CRAN (R 4.1.2)
 pkgconfig        2.0.3      2019-09-22 [1] CRAN (R 4.0.4)
 plyr             1.8.6      2020-03-03 [1] CRAN (R 4.0.4)
 posterior        1.2.0      2022-01-05 [1] CRAN (R 4.1.2)
 prettyunits      1.1.1      2020-01-24 [1] CRAN (R 4.0.4)
 processx         3.5.2      2021-04-30 [1] CRAN (R 4.0.5)
 promises         1.2.0.1    2021-02-11 [1] CRAN (R 4.0.4)
 ps               1.6.0      2021-02-28 [1] CRAN (R 4.0.4)
 purrr            0.3.4      2020-04-17 [1] CRAN (R 4.0.4)
 R6               2.5.1      2021-08-19 [1] CRAN (R 4.1.1)
 Rcpp           * 1.0.8      2022-01-13 [1] CRAN (R 4.1.2)
 RcppParallel     5.1.5      2022-01-05 [1] CRAN (R 4.1.2)
 reshape2         1.4.4      2020-04-09 [1] CRAN (R 4.0.4)
 rlang            1.0.1      2022-02-03 [1] CRAN (R 4.1.2)
 rsconnect        0.8.25     2021-11-19 [1] CRAN (R 4.1.2)
 rstan            2.26.6     2021-12-15 [1] local
 rstantools       2.1.1      2020-07-06 [1] CRAN (R 4.0.4)
 sandwich         3.0-1      2021-05-18 [1] CRAN (R 4.1.0)
 scales           1.1.1      2020-05-11 [1] CRAN (R 4.0.4)
 sessioninfo      1.2.2      2021-12-06 [1] CRAN (R 4.1.2)
 shiny            1.7.1      2021-10-02 [1] CRAN (R 4.1.1)
 shinyjs          2.1.0      2021-12-23 [1] CRAN (R 4.1.2)
 shinystan        2.5.0      2018-05-01 [1] CRAN (R 4.0.4)
 shinythemes      1.2.0      2021-01-25 [1] CRAN (R 4.0.4)
 StanHeaders      2.26.6     2021-12-15 [1] local
 stringi          1.7.6      2021-11-29 [1] CRAN (R 4.1.2)
 stringr          1.4.0      2019-02-10 [1] CRAN (R 4.0.4)
 survival         3.2-13     2021-08-24 [4] CRAN (R 4.1.1)
 tensorA          0.36.2     2020-11-19 [1] CRAN (R 4.0.4)
 testthat       * 3.1.2      2022-01-20 [1] CRAN (R 4.1.2)
 TH.data          1.1-0      2021-09-27 [1] CRAN (R 4.1.1)
 threejs          0.3.3      2020-01-21 [1] CRAN (R 4.0.4)
 tibble           3.1.6      2021-11-07 [1] CRAN (R 4.1.2)
 tidyselect       1.1.1      2021-04-30 [1] CRAN (R 4.0.5)
 utf8             1.2.2      2021-07-24 [1] CRAN (R 4.1.0)
 V8               4.1.0      2022-02-06 [1] CRAN (R 4.1.2)
 vctrs            0.3.8      2021-04-29 [1] CRAN (R 4.0.5)
 xtable           1.8-4      2019-04-21 [1] CRAN (R 4.0.4)
 xts              0.12.1     2020-09-09 [1] CRAN (R 4.0.4)
 zoo              1.8-9      2021-03-09 [1] CRAN (R 4.0.4)

 [1] <path_to_primary_lib>
 [2] /usr/local/lib/R/site-library
 [3] /usr/lib/R/site-library
 [4] /usr/lib/R/library

────────────────────────────────────────────────────────

Note: rstan and StanHeaders are from https://mc-stan.org/r-packages/.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked tstbrmskfold minimal R package and repeatedly run R CMD check to observe the intermittent failure. Compare the rstan-backed brms/loo::kfold() behavior with the cmdstanr backend; done means identifying the responsible component and establishing a deterministic reproduction or documenting why no fix is possible.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.