stan-dev / stan-dev/projpred

Forward search without candidates at specific model size

Open
#307 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

perhaps
Dominant language
R
Stars
114
Forks
31
PR merge metrics
No merged PRs in 30d

Description

I'm currently working on the search_terms argument (fixing bugs and improving documentation). While doing so, I realized that there can be model sizes for which the forward search doesn't have any candidate models, for example:

options(mc.cores = parallel::detectCores(logical = FALSE))
data("df_gaussian", package = "projpred")
df_gaussian <- df_gaussian[1:41, ]
dat <- data.frame(y = df_gaussian$y, df_gaussian$x)
library(rstanarm)
rfit <- stan_glm(y ~ X1 + X2 + X3 + X4 + X5,
                 data = dat,
                 seed = 1140350788)
library(projpred)
vs <- varsel(rfit,
             nclusters = 3,
             nclusters_pred = 5,
             method = "forward",
             search_terms = c("X1 + X2"),
             seed = 46782345)

(tested with projpred 2.1.1). If you inspect the output of that varsel() call, you'll see that X1 + X2 is regarded as the solution term at model size 1:

print(vs)

gives


Family: gaussian 
Link function: identity 

Formula: y ~ X1 + X2 + X3 + X4 + X5
Observations: 41
Search method: forward, maximum number of terms 1
Number of clusters used for selection: 3
Number of clusters used for prediction: 5
Suggested Projection Size: NA

Selection Summary:
 size solution_terms   elpd  se  diff diff.se
    0           <NA> -101.6 2.9 -17.4     3.4
    1        X1 + X2  -93.9 2.8  -9.7     2.3

and plot(vs) behaves accordingly. Now my question (especially to @AlejandroCatalina) is whether this is intended or whether X1 + X2 should be regarded as the solution term at model size 2 because it consists of the 2 terms X1 and X2. The latter would probably require some larger changes because all functions downstream of search_forward() would have to be adapted to deal with "empty model sizes".

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproducible R example in the issue and inspect search_forward(), then trace how varsel() and plot(vs) consume its model sizes. First determine whether a grouped search term belongs at its term count or at the next model size. Done means the chosen behavior is consistent in the selection summary and plot, including sizes with no candidate models.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.