Make it easier for user to create benchmark suites
Open
@joaquinvanschoren is already working on this.
Since Mar 15, 2018.
- Dominant language
- PHP
- Stars
- 755
- Forks
- 128
- PR merge metrics
- No merged PRs in 30d
Description
Originally posted by @giuseppec
- We need to describe how to create a suite better. Currently, we have this https://www.openml.org/guide/benchmark . Maybe we can add something like this (still improvable):
- To create a benchmark suite, we need to use tasks (not datasets). That is, if there is no task for the corresponding dataset, you have to first create a task out of it (see https://www.openml.org/new/task which is currently only possible through the web interface).
- You have to create a study https://www.openml.org/new/study (I think this is currently also only possible through the web interface) and remember the study ID after you have created the study, you will need the ID for step c). If you set an alias-string when creating the study, it can then also be used to retrieve the benchmark suite (alternatively the study ID can be used, see step d).
- You should add a tag called "study_X" where X = your study ID to the tasks (and datasets), this should be possible by the clients (e.g. R) or through web interface.
- Now you have your benchmark suite. In R, you can get the information using
getOMLStudy(IDofStudy)orgetOMLStudy("your-alias-string"). Study information can be found online https://www.openml.org/s/IDofStudy
- We maybe have to simplify some steps for users. Still, many things are only possible through the web interface. Example:
- we need a better way to create tasks out of datasets, imagine you want to add your one benchmark suite but have to create 100 tasks manually through the web. See https://github.com/openml/OpenML/issues/325
- If a task is tagged, also the underlying data should be tagged. Also if a run is tagged, then the underlying task, data and flow should be tagged by the same tag. See https://github.com/openml/OpenML/issues/530 . If the server does not do this automatically, at least the client should do this.
- Maybe we should also allow that tagging tasks by alias-string are also accepted in step (c) above.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.