stan-dev / stan-dev/projpred

Unexpected behaviour when incompatible full formulas included in search_terms argument

Open
#362 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
R
Stars
114
Forks
31
PR merge metrics
No merged PRs in 30d

Description

Hi,

I noticed that when full formulas are used in search_terms input argument such that some formulas are incompatible, cv_varsel doesn't check the incompatibility and includes variables from two incompatible formulas. This can be seen in the following reprex

library(brms)
library(projpred)
set.seed(2)
N <- 100
p <- 5 #number of parameters
dat <- as.data.frame(matrix(rnorm(N*p), nrow = N, ncol = p)) #initialize data frame with p covariates
names(dat) <- paste0('x', 1:p)

betas <- rnorm(p) # simulate effect values
dat$y <- rnorm(N, mean=as.matrix(dat[, paste0('x', 1:p)]) %*% betas) #y is a noisy observation of the linear combination of covariates

formula_all <- as.formula(paste0('y~', paste(paste0('x', 1:p), collapse = '+')))
ref_mod <- brm(formula_all, data = dat, refresh = 0)
cv_select_prj <- cv_varsel(ref_mod, method = 'forward', cv_method = 'LOO', refit_prj = F, search_terms = c('x1+x2','x1+x3','x1+x2+x4'))

Here, I would assume that either x1, x2, x4 would be included or x1 and x3 only, as x1+x2+x4 is not compatible with x1+x3. However, cv_varsel returns:

Selection Summary:
 size solution_terms elpd.loo  se  diff diff.se
    0           <NA>   -181.9 6.0 -36.8     7.0
    1        x1 + x3   -161.3 6.0 -16.2     5.3
    2             x2   -149.1 6.1  -4.1     3.1
    3             x4   -147.4 5.8  -2.3     2.5

Is this the intended behaviour or should we try to resolve this? I The problem seems to lie in select_possible_terms_size in formula.R.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reprex in the issue and inspect select_possible_terms_size in formula.R, where incompatible full formulas are reportedly accepted. Determine the intended handling of x1+x2+x4 versus x1+x3, then verify that cv_varsel no longer combines incompatible terms while preserving valid selections. The issue is old and has no linked pull request or recent discussion.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.