JuliaAI / JuliaAI/StatisticalMeasures.jl
Probabilistic measures with Monte Carlo draws
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 23
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
Sometimes we don't have a density function for a predictive distribution but instead have IID draws from that distribution. e.g. in Bayesian inference, we often only have access to the posterior predictive distribution through Monte Carlo draws. It would be useful if this package supported statistical estimation of the measure in such cases.
Proposal
I propose the API for probabilistic measures be expanded so that the first argument to a probabilistic measure can be an array of predictions representing these draws instead of just a distribution, where possible (e.g. this would not be possible for LogScore). When the first argument is an array of draws, then weights must be an array with the same size.
Example: CRPS
For example, the continuously ranked probability score (CRPS) can be defined for a predictive distribution q as
CRPS(q, y) = E[|ŷ-y|] - E[|ŷ-ŷ'|], where ŷ' and ŷ are independent random variables drawn from q.
For many named distributions, this can be evaluated exactly. But it can also be estimated from draws.
The naive estimator of CRPS for an IID sample of size N is.
mean(abs.(ŷ .- y)) - mean(abs.(ŷ .- ŷ')) * (N // (2 * (N-1))). For continuous predictions, this can be reformulated using order statistics to reduce the cost to O(Nlog(N)) (see https://hal.science/hal-02976423/document)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing probabilistic-measure entry points and the CRPS implementation, then compare them with the proposal's array-of-draws API and weighting requirements. Done means supported measures accept draw arrays with same-sized weights where possible, while unsupported cases such as LogScore remain defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100