arviz-devs / arviz-devs/PosteriorStats.jl

Additional LOO utilities

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Julia
Stars
11
Forks
1
PR merge metrics
No merged PRs in 30d

Description

From https://github.com/arviz-devs/arviz/issues/2059, we should add the following utilities:
- `loo_expectation(log_likelihood, expectand; kwargs...)`: Approximate each `expectand` with respect to each LOO posterior using PSIS-LOO. e.g. `loo_pit` could be reimplemented by setting `expectand = y_pred .≤ reshape(y, 1, 1, size(y)...)`. It's important to support the option of computing expectation-specific PSIS diagnostics, which requires https://github.com/arviz-devs/PSIS.jl/issues/21.
- `loo_predictive_error(log_likelihood, metric; kwargs...)`: uses `loo_expectation` to compute LOO posterior predictive means and then use `metric` to evaluate the quality of the fit.
- `loo_crps(psis_loo_result; scale=false, kwargs...)`: compute continuous ranked probability score (CRPS) or it's scaled variant SCRPS, which is another strictly proper scoring rule besides Log score (ELPD) to use for model comparison. Requires 2 posterior draws for each draw-chain. But before adding this, we should think about the model comparison API some more. In principle the user could select different scores or accuracy metrics for model comparison. And section 2.1 of https://arxiv.org/abs/1704.02030 generalizes stacking weights for any proper scoring rule, so our stacking implementation could be updated to do so as well.

The APIs written here are putative. Odds are they will be refined during implementation.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the proposed loo_expectation, loo_predictive_error, and loo_crps APIs, along with the linked PSIS.jl issue 21. Then examine the existing loo_pit and stacking implementation entry points and clarify the model-comparison API and scoring-rule scope. Done requires an agreed, testable scope before implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
analytics, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.