No way to submit prediction uncertainties that are not probabilities
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 755
- Forks
- 128
- PR merge metrics
- No merged PRs in 30d
Description
From talking with @mfeurer it seems like there's no way to submit continuous predictions for classification that are not probabilities.
I think that's an issue because you can compute binary AUC or OVR AUC without probabilities.
Not using decision values leads to inconsistencies (the AUC of an SVM is much lower in openml than if computed with sklearn because the openml computation is wrong).
Also see https://github.com/openml/openml-python/issues/786
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the prediction-submission behavior described here and the related openml-python issue #786. Determine how continuous classification predictions are currently represented and how AUC is computed, then define what support for non-probability decision values would require and how correctness would be verified against sklearn.
Written by the indexing model from the issue text.
Assessment
- Domain
- api, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100