stan-dev / stan-dev/rstanarm

Draw order inconsistent with `posterior_linpred` and `posterior_epred`

Open
#622 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
401
Forks
136
PR merge metrics
No merged PRs in 30d

Description

Summary:

It appears that every invocation of posterior_linpred and posterior_epred gives a different permutation of the draws. This prevents users from doing posterior calculations (e.g. covariances) using the draws.

Description:

It can be useful to use the output of posterior_linpred and posterior_epred in further computation, e.g., as part of covariances with model parameters. As far as I can tell this is currently impossible, since each invocation seems to return a different random permutation of the draws.

Reproducible Steps:
# Simulate some logistic regression data
set.seed(42)

n_obs <- 1000

num_cat <- 3
rel_min <- 0.1
cat_probs <- runif(num_cat) 
cat_probs[cat_probs < rel_min] <- rel_min
cat_probs <- cat_probs / sum(cat_probs)
x <- sample(num_cat, size=n_obs, replace=TRUE, prob=cat_probs)
y_prob <- runif(num_cat)

df <-
  data.frame(x=x, p=y_prob[x]) %>%
  mutate(y=as.numeric(runif(n()) < p))


num_draws <- 5000

# Run rstanarm

  fit <- stan_glmer(y ~ (1 | x),
                    family = binomial(link = "logit"),
                    data = df,
                    prior = normal(0, 1, autoscale = TRUE),
                    prior_covariance = decov(scale = 0.50),
                    adapt_delta = 0.99,
                    refresh = 0,
                    seed = 1010,
                    iter = num_draws)

# Then every one of these repeated commands gives a different answer each time:
  posterior_linpred(fit, newdata=df, draws=num_draws, seed=42, permuted=FALSE)[1,1]  
  posterior_linpred(fit, newdata=df, draws=num_draws, seed=42, permuted=FALSE)[1,1]  
  posterior_linpred(fit, newdata=df, draws=num_draws, seed=42)[1,1]  
  posterior_linpred(fit, newdata=df, draws=num_draws, seed=42)[1,1]  
  posterior_linpred(fit, newdata=df, draws=num_draws)[1,1]  
  posterior_linpred(fit, newdata=df, draws=num_draws)[1,1]  
  posterior_linpred(fit, newdata=df, draws=num_draws - 10)[1,1]  
  posterior_linpred(fit, newdata=df, draws=num_draws - 10)[1,1]
RStanARM Version:

rstanarm Version: 2.21.4

R Version:

4.1.2

Operating System:

Operating System: Ubuntu 22.04.4 LTS

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the behavior through the posterior_linpred and posterior_epred entry points using the R example in the issue. Trace how draws and seeds are selected across repeated calls, then verify that related posterior calculations preserve draw correspondence and that the provided reproducible commands no longer return inconsistent permutations.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.