arviz-devs / arviz-devs/PosteriorStats.jl
Adding utilities and docs for estimating ELPD through model refitting
- Dominant language
- Julia
- Stars
- 11
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
[`arviz.reloo`](https://python.arviz.org/en/stable/api/generated/arviz.reloo.html#arviz.reloo) allows for drawing samples from a LOO posterior for all data points for which the PSIS-LOO pareto shape diagnostic indicates that PSIS-LOO gives poor estimates.
We should think carefully about how to support this. In particular, we need to design an API that can be easily extended by e.g. a PPL but is straightforward to use without a PPL. The result should be an `AbstractELPDResult`, but we need some way to indicate that this elpd result comes from different estimation methods. Hence the LOO posteriors refitted should probably have a `missing` pareto shape diagnostic.
But more generally, once we have this functionality, running exact LOO, LOGO, or k-fold CV should be possible, so the API should support that as well.
In addition, the functionality in #8 should be supported, but this is more complicated, because posterior predictions may be required, and because the order of operations is reversed. With PSIS-LOO, we compute approximate LOO once and reuse the importance weights for all expectations from LOO posteriors, while with exact LOO, one loops over each data point held out and computes each expectation for each LOO posterior.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the arviZ.reloo API referenced in the issue and reviewing the functionality described in #8. Design an extensible API for refitting-based ELPD methods, including exact LOO, LOGO, and k-fold CV, with results represented as AbstractELPDResult and a way to distinguish estimation methods; done means the documented use cases are supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100