openml / openml/OpenML

Dataset sql schema

Open
#826 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

CoreSystem
Dominant language
PHP
Stars
755
Forks
128
PR merge metrics
No merged PRs in 30d

Description

Inspired by #825

Looking at the following database schema, there seems to be some undocumented properties. Cross check with XSD shows the following doubts:

CREATE TABLE `dataset` (
  `did` int(10) UNSIGNED NOT NULL,
  `uploader` int(10) DEFAULT NULL,
  `source` int(10) UNSIGNED DEFAULT NULL,                                       # JvR: clearly not used.
  `name` varchar(128) COLLATE utf8_unicode_ci NOT NULL,
  `version` varchar(64) COLLATE utf8_unicode_ci NOT NULL,
  `version_label` varchar(128) COLLATE utf8_unicode_ci DEFAULT NULL,            # JvR: What was the version label again? XSD does not display this
  `description` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `format` varchar(64) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'arff',
  `creator` text COLLATE utf8_unicode_ci,
  `contributor` text COLLATE utf8_unicode_ci,
  `collection_date` varchar(128) COLLATE utf8_unicode_ci DEFAULT NULL,
  `upload_date` datetime NOT NULL,
  `language` varchar(128) COLLATE utf8_unicode_ci DEFAULT NULL,
  `licence` varchar(64) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'Public',
  `citation` text COLLATE utf8_unicode_ci,                                            # JvR: what is difference with paper_url? 
  `collection` varchar(64) CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL,          # JvR: not used anymore
  `url` mediumtext CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,                      # JvR: not used anymore, can be removed. 
  `isOriginal` enum('true','false') CHARACTER SET utf8 COLLATE utf8_bin DEFAULT NULL, # JvR: not sure if used
  `file_id` int(10) DEFAULT NULL,
  `default_target_attribute` varchar(1024) COLLATE utf8_unicode_ci DEFAULT NULL,
  `row_id_attribute` varchar(128) COLLATE utf8_unicode_ci DEFAULT NULL,
  `ignore_attribute` varchar(128) COLLATE utf8_unicode_ci DEFAULT NULL,
  `paper_url` mediumtext COLLATE utf8_unicode_ci,                               # JvR: clearly useful, but not sure if used?
  `visibility` varchar(128) COLLATE utf8_unicode_ci NOT NULL DEFAULT 'public',
  `original_data_id` int(10) DEFAULT NULL,                                      # JvR: not sure if used?
  `original_data_url` mediumtext COLLATE utf8_unicode_ci,                       # JvR: not sure if used?
  `update_comment` text COLLATE utf8_unicode_ci,                                # JvR: not sure if used?
  `last_update` datetime DEFAULT NULL                                           # JvR: not sure if used?
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

We probably want to figure out which of the seemingly unused fields can be removed, to keep the system simple. Also, we should add something that facilitates to show dataset dependencies when the names do not collide, as for example with adult and adult-census in #794 .

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the listed dataset SQL schema with the referenced openml.data.upload.xsd, then trace whether the questioned fields are used. Review #794 for the dataset dependency problem and #825 for context. Done means identifying safe removals and defining how dependencies such as adult and adult-census should be represented.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.