openml / openml/server-api

Evaluation measures and task types should fit together

Open
#334 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug php
Dominant language
Python
Stars
16
Forks
50
PR merge metrics
No merged PRs in 30d

Description

Example: There are tasks on OpenML that have as task type Supervised regression and as Evalutation measure predictive accuracy (e.g. http://www.openml.org/t/2994). Should this be allowed?

library("OpenML")

alltasks <- listOMLTasks()

tasks_supervpred <- alltasks[alltasks$task.type %in% "Supervised Regression" & alltasks$evaluation.measures %in% "predictive_accuracy",]
runevals_supervpred <- lapply(tasks_supervpred$task.id, function(tid) {
  message(tid)
  try(listOMLRunEvaluations(task.id = tid))
  }) 

Task ids are:

> tasks_supervpred$task.id
  [1]  2994  2995  2996  2997  2998  2999  3000  3001  3002  3003  3004  3005  3006  3007  3008  3009  5482  5483  5484  5485  5486  5487  5488  5489  5490
 [26]  5491  5492  5493  5494  5495  5496  5497  5498  5499  5500  5501  5502  5503  5504  5505  5506  5507  5508  5509  5510  5511  5512  5513  5514  5515
 [51]  5516  5517  7318  7319  7320  7321  7322  7323  7324  7325  7326  7327  7393  7561  7564  7565  7566  7567  7568  7569  7570  7571  7572  7573  7574
 [76]  7575 10102 12714 12715 12716 12717 12718 12719 12720 12721 12722 12723 12724 12725 12726 12727 12728 12729 12730 12731 12732 12733 12734 12735 14950

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the behavior with the listed OpenML task IDs and the R calls in the issue. Then trace the server-side validation or handling of task types and evaluation measures. Done means the relationship between supervised regression tasks and predictive_accuracy is explicitly resolved and handled consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, r
Domain
api, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.