JuliaAI / JuliaAI/StatisticalMeasures.jl

Probabilistic measures with Monte Carlo draws

Open
#36 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
23
Forks
7
PR merge metrics
No merged PRs in 30d

Description

Sometimes we don't have a density function for a predictive distribution but instead have IID draws from that distribution. e.g. in Bayesian inference, we often only have access to the posterior predictive distribution through Monte Carlo draws. It would be useful if this package supported statistical estimation of the measure in such cases.

Proposal

I propose the API for probabilistic measures be expanded so that the first argument to a probabilistic measure can be an array of predictions representing these draws instead of just a distribution, where possible (e.g. this would not be possible for LogScore). When the first argument is an array of draws, then weights must be an array with the same size.

Example: CRPS

For example, the continuously ranked probability score (CRPS) can be defined for a predictive distribution q as
CRPS(q, y) = E[|ŷ-y|] - E[|ŷ-ŷ'|], where ŷ' and are independent random variables drawn from q.
For many named distributions, this can be evaluated exactly. But it can also be estimated from draws.

The naive estimator of CRPS for an IID sample of size N is.
mean(abs.(ŷ .- y)) - mean(abs.(ŷ .- ŷ')) * (N // (2 * (N-1))). For continuous predictions, this can be reformulated using order statistics to reduce the cost to O(Nlog(N)) (see https://hal.science/hal-02976423/document)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing probabilistic-measure entry points and the CRPS implementation, then compare them with the proposal's array-of-draws API and weighting requirements. Done means supported measures accept draw arrays with same-sized weights where possible, while unsupported cases such as LogScore remain defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.