Broadcasting probabilistic `Unaggregated` measures (such as proper scoring rules) over a single prediction
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 163
- Forks
- 46
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 5
Description
The idea of implementing "distribution fitters" as supervised models is to enable their evaluation using a proper scoring rule (which in turn makes hyper-parameter optimization possible).
There are precisely zero models implementing this API, but as currently specified, predict(mach, nothing) returns the fitted distribution. Now to evaluate using some examples using, say, log_score, we need a broadcast version of log_score(yhat, y) where yhat is a single distribution and y a number of observations (samples). Currently measures expect yhat and y to be arrays of the same dimension.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the measure API around log_score(yhat, y) and the supervised-model behavior where predict(mach, nothing) returns a fitted distribution. Determine the intended semantics for evaluating one distribution against multiple observations, then define and test the broadcasting behavior so scoring such samples is covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100