arviz-devs / arviz-devs/PosteriorStats.jl

Adding utilities and docs for estimating ELPD through model refitting

Open
#12 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
11
Forks
1
PR merge metrics
No merged PRs in 30d

Description

[`arviz.reloo`](https://python.arviz.org/en/stable/api/generated/arviz.reloo.html#arviz.reloo) allows for drawing samples from a LOO posterior for all data points for which the PSIS-LOO pareto shape diagnostic indicates that PSIS-LOO gives poor estimates.

We should think carefully about how to support this. In particular, we need to design an API that can be easily extended by e.g. a PPL but is straightforward to use without a PPL. The result should be an `AbstractELPDResult`, but we need some way to indicate that this elpd result comes from different estimation methods. Hence the LOO posteriors refitted should probably have a `missing` pareto shape diagnostic.

But more generally, once we have this functionality, running exact LOO, LOGO, or k-fold CV should be possible, so the API should support that as well.

In addition, the functionality in #8 should be supported, but this is more complicated, because posterior predictions may be required, and because the order of operations is reversed. With PSIS-LOO, we compute approximate LOO once and reuse the importance weights for all expectations from LOO posteriors, while with exact LOO, one loops over each data point held out and computes each expectation for each LOO posterior.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the arviZ.reloo API referenced in the issue and reviewing the functionality described in #8. Design an extensible API for refitting-based ELPD methods, including exact LOO, LOGO, and k-fold CV, with results represented as AbstractELPDResult and a way to distinguish estimation methods; done means the documented use cases are supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.