stan-dev / stan-dev/loo

problems using loo in clogit models

Open
#44 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
157
Forks
38
Avg merge
4d 16h
Merged PRs (30d)
2

Description

In a rstanarm branch, I introduced a stan_clogit function which is similar to the clogit function in the survival package but is actually described in more detail by Stata ( http://www.stata.com/manuals13/rclogit.pdf ). So, you will be able to do

post <- stan_clogit(case ~ spontaneous + induced, strata = stratum,
                            data = infert[order(infert$stratum, !infert$case),], QR = TRUE)

The problem is that some groups have the same values on the dummy variables spontaneous and induced, such as

> infert[infert$stratum == 3,]
    education age parity induced case spontaneous stratum pooled.stratum
3      0-5yrs  39      6       2    1           0       3              4
86     0-5yrs  39      6       2    0           0       3              4
168    0-5yrs  39      6       2    0           0       3              4

Thus, no matter what are the posterior realizations of the coefficients on spontaneous and induced, the likelihood for group 3 is the same.

I am pretty sure the correct concept for a clogit model is to imagine leaving out one group rather than one observation, but when I call loo::loo.matrix, the third column of the input is a constant, which causes the Pareto k estimate to be infinite. Since the third group could be omitted and only change the log-likelihood by a constant, this seems to be unreasonable.

The question becomes, what function should be catching this? We could have stan_clogit drop groups that have only one unique row in the design matri{x,ces}. We could have loo.stanreg omit such groups. Or loo::loo could check which Pareto k estimates are infinite and change them to some number when the log likelihoods are finite but constant. Thoughts @avehtari ?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the constant third-column Pareto-k behavior with loo::loo.matrix using the stan_clogit example and infert data described in the issue. Review how loo::loo, loo::loo.matrix, and loo.stanreg divide responsibility, then resolve which behavior is intended and define regression coverage for the agreed handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.