Evaluation measures and task types should fit together
Open
Nobody has claimed this yet.
bug
php
- Dominant language
- Python
- Stars
- 16
- Forks
- 50
- PR merge metrics
- No merged PRs in 30d
Description
Example: There are tasks on OpenML that have as task type Supervised regression and as Evalutation measure predictive accuracy (e.g. http://www.openml.org/t/2994). Should this be allowed?
library("OpenML")
alltasks <- listOMLTasks()
tasks_supervpred <- alltasks[alltasks$task.type %in% "Supervised Regression" & alltasks$evaluation.measures %in% "predictive_accuracy",]
runevals_supervpred <- lapply(tasks_supervpred$task.id, function(tid) {
message(tid)
try(listOMLRunEvaluations(task.id = tid))
})
Task ids are:
> tasks_supervpred$task.id
[1] 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 5482 5483 5484 5485 5486 5487 5488 5489 5490
[26] 5491 5492 5493 5494 5495 5496 5497 5498 5499 5500 5501 5502 5503 5504 5505 5506 5507 5508 5509 5510 5511 5512 5513 5514 5515
[51] 5516 5517 7318 7319 7320 7321 7322 7323 7324 7325 7326 7327 7393 7561 7564 7565 7566 7567 7568 7569 7570 7571 7572 7573 7574
[76] 7575 10102 12714 12715 12716 12717 12718 12719 12720 12721 12722 12723 12724 12725 12726 12727 12728 12729 12730 12731 12732 12733 12734 12735 14950
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the behavior with the listed OpenML task IDs and the R calls in the issue. Then trace the server-side validation or handling of task types and evaluation measures. Done means the relationship between supervised regression tasks and predictive_accuracy is explicitly resolved and handled consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, r
- Domain
- api, backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100