openml / openml/OpenML

Default target for multi task datasets

Open
#799 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

CoreSystem
Dominant language
PHP
Stars
755
Forks
128
PR merge metrics
No merged PRs in 30d

Description

Looking at the XSD, it occurred to me that the following field has some problems:
xs:element name="default_target_attribute" minOccurs="0" type="oml:casual_string1024"/> <!-- The default target attribute, if it exists. Can also have multiple values (comma-separated). Of course, tasks can be defined that use another attribute as target -->

I would propose to change this to:
xs:element name="default_target_attribute" minOccurs="0" max_occurs="unbounded" type="oml:casual_string1024"/> <!-- The default target attribute, if it exists.-->

For the following reasons:

  • Comma separated is a suboptimal solution. I wouldn't be surprised if a comma is a valid entry in an arff attribute, which makes it impossible to parse several arff headers
  • This distributes the burden of parsing to the specific libraries, java, weka, python, mlr and the evaluation engine all have to deal with this
  • This currently requires quite a hack in the evaluation engine
  • it is inconsistent with other fields, for example: creator, contributor, ignore attribute, etc.

Main question is how to solve this.
a) which libraries do depend on the feature that we can mark and upload this as a comma separated string? (AFAIK java and python not, but maybe mlr does)
b) How to fix the current fields in the database?
c) evaluation engine has hacked around this behavior, I would like to remove these hacks and set everything in such a way that it consistent with the other fields
d) ...

@joaquinvanschoren @berndbischl @mfeurer @giuseppec

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with openml_OS/views/pages/api_new/v1/xsd/openml.data.upload.xsd and trace how default_target_attribute is consumed by the Java, Weka, Python, and mlr libraries and the evaluation engine. Review the existing database fields and evaluation-engine workaround. Done means the compatibility impact, database migration, and consistent handling of multiple target attributes are agreed and documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
xml
Domain
backend-api-design, databases, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.