Internal Task Format

Looking at the current OpenML internal task implementation, I encountered several problems.

• The database model is sub-optimal
• As one table facilitates now the storage of all task types with all possible values, there are currently no database integrity checks on the data (in fact, the value field in the database is a TEXT value of at most 64k characters)
• This makes indexing complicated
• This relies completely on my PHP integrity checks (which are due to the almost free for all structure pretty ad-hoc)
• I already found some inconsistencies on data level (hopefully/presumably these were introduced before I implemented the integrity checks).

This is the current database table scheme:
task_inputs

• task_id (int, fk to table task)
• input (varchar, not really a real foreign key but somewhat related to table task_io_types)
• value (text, free for all)
  (PK: fields task_id and input)

I would propose the following changes to the structure:

• make field input a true foreign key to the table task_io_types (and insert an int reference instead of textual reference)
• make value field an integer value (unfortunately, we can't allow foreign keys here without exploding the number of tables)

Especially, the last will have serious consequences. First of all, all current task inputs need to be converted to integer format (i.e., a key in another table). Second, some of them may not be able to be converted. For example, the custom_holdoutset. I never liked this feature anyway, it's not well-tested, and barely used. The following query shows the tasks that do make use of this feature, which are almost all (deactivated) datasets.

SELECT task_id, COUNT(*) FROM run WHERE task_id IN (SELECT task_id FROM `task_inputs` WHERE input = "custom_testset") GROUP BY task_id ORDER BY COUNT(*)