arviz-devs / arviz-devs/PosteriorStats.jl
Additional LOO utilities
- Dominant language
- Julia
- Stars
- 11
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
From https://github.com/arviz-devs/arviz/issues/2059, we should add the following utilities:
- `loo_expectation(log_likelihood, expectand; kwargs...)`: Approximate each `expectand` with respect to each LOO posterior using PSIS-LOO. e.g. `loo_pit` could be reimplemented by setting `expectand = y_pred .≤ reshape(y, 1, 1, size(y)...)`. It's important to support the option of computing expectation-specific PSIS diagnostics, which requires https://github.com/arviz-devs/PSIS.jl/issues/21.
- `loo_predictive_error(log_likelihood, metric; kwargs...)`: uses `loo_expectation` to compute LOO posterior predictive means and then use `metric` to evaluate the quality of the fit.
- `loo_crps(psis_loo_result; scale=false, kwargs...)`: compute continuous ranked probability score (CRPS) or it's scaled variant SCRPS, which is another strictly proper scoring rule besides Log score (ELPD) to use for model comparison. Requires 2 posterior draws for each draw-chain. But before adding this, we should think about the model comparison API some more. In principle the user could select different scores or accuracy metrics for model comparison. And section 2.1 of https://arxiv.org/abs/1704.02030 generalizes stacking weights for any proper scoring rule, so our stacking implementation could be updated to do so as well.
The APIs written here are putative. Odds are they will be refined during implementation.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the proposed loo_expectation, loo_predictive_error, and loo_crps APIs, along with the linked PSIS.jl issue 21. Then examine the existing loo_pit and stacking implementation entry points and clarify the model-comparison API and scoring-rule scope. Done requires an agreed, testable scope before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- analytics, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100