stan-dev / stan-dev/loo

Moment matching with iwmm package

Open
#212 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
157
Forks
38
Avg merge
4d 16h
Merged PRs (30d)
2

Description

Currently loo_moment_match only works on stanfit objects (due to the reliance on the log_prob methods from rstan), but not CmdStanFit or just a matrix of draws (see e.g. #209). I was thinking about the best way to expand support of loo_moment_match to other objects, and am opening this issue mostly for discussion at this stage.

@topipa has created a generic implementation of moment matching (iwmm) which works on a matrix or stanfit object. Recently we have also added CmdStanFit support to this package (using the new model methods in cmdstanr which are currently not yet in a release).

iwmm is not (yet) on CRAN, so now might be a good time to discuss the best way to interoperate with loo.
After some discussion with @avehtari, I currently see three options (there may be others I haven't thought of):

  1. update functions in loo, no change to dependencies
    Code from iwmm is copied into loo and adapted for use in loo::loo_moment_match (made specific for leave-one-out importance posteriors). iwmm would remain an independent package for generic importance sampling.
  2. add iwmm dependency in loo
    iwmm is submitted to CRAN and loo::loo_moment_match is rewritten to use iwmm::moment_match. loo would then import and depend on iwmm.
  3. move iwmm functions into loo
    All iwmm functions are moved into loo, and kept generic (i.e. not specific for leave-one-out posteriors). Keeping the functions generic has some precedence as loo is the home of loo::psis which is used in cases other than leave-one-out CV (e.g. in priorsense and adjustr).

@jgabry @avehtari @topipa @paul-buerkner , do you have any thoughts on this?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing loo::loo_moment_match and the generic implementation in the linked iwmm package, including its matrix, stanfit, and CmdStanFit support. Compare the three interoperability options in the issue and determine which design should be adopted; done means the project has a decided approach for expanding supported inputs.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.