biolab / biolab/orange2

Standard deviation or other measures of error for AUC

Open
#31 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
304
Forks
142
PR merge metrics
No merged PRs in 30d

Description

Is there a way to get some kind of error measurement (eg: standard deviation, confidence interval) of the AUC after doing cross validations? It would be helpful to get a list of all such values from doing cross validations instead of just a single AUC.

The relevant functions are AUC(test_results=None, multiclass=0, ignore_weights=False) and cross_validation(learners, examples, folds=10, stratified=StratifiedIfPossible, preprocessors=(), random_generator=0, callback=None, store_classifiers=False, store_examples=False).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the AUC(test_results=None, multiclass=0, ignore_weights=False) and cross_validation(learners, examples, folds=10, ...) entry points named in the issue. Trace how cross-validation results are collected and determine where per-fold AUC values and error measures would be exposed. Done means cross-validation can return the requested values alongside the single AUC, with behavior clarified for multiclass and weighted cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.