Mean and mean_weighted measures are not computed
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 755
- Forks
- 128
- PR merge metrics
- No merged PRs in 30d
Description
For multi-class classification tasks, measures such mean_weighted_area_under_roc_curve, mean_precision, and mean_weighted_precision are not computed. Instead, area_under_roc_curve and precision are already weighted by class size. This is inconsistent with the evaluation measure descriptions (https://www.openml.org/search?type=measure) and also means that you cannot actually choose between e.g. weighted and unweighted precision. Is this intentional?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the multi-class evaluation code that computes area_under_roc_curve and precision, then compare its behavior with the named mean and mean_weighted measures and the linked evaluation-measure descriptions. Done means determining whether the measures should be computed separately and documenting or correcting the behavior so weighted and unweighted precision can be distinguished.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- php
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100