openml / openml/OpenML

Mean and mean_weighted measures are not computed

Open
#627 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PHP
Stars
755
Forks
128
PR merge metrics
No merged PRs in 30d

Description

For multi-class classification tasks, measures such mean_weighted_area_under_roc_curve, mean_precision, and mean_weighted_precision are not computed. Instead, area_under_roc_curve and precision are already weighted by class size. This is inconsistent with the evaluation measure descriptions (https://www.openml.org/search?type=measure) and also means that you cannot actually choose between e.g. weighted and unweighted precision. Is this intentional?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the multi-class evaluation code that computes area_under_roc_curve and precision, then compare its behavior with the named mean and mean_weighted measures and the linked evaluation-measure descriptions. Done means determining whether the measures should be computed separately and documenting or correcting the behavior so weighted and unweighted precision can be distinguished.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.