Provide aggregated performance info for flows
Nobody has claimed this yet.
- Dominant language
- PHP
- Stars
- 755
- Forks
- 128
- PR merge metrics
- No merged PRs in 30d
Description
It could be interesting to store aggregated scores for various evaluation metrics per flow, e.g. as a property (quality) of the flow:
Also see https://github.com/openml/OpenML/issues/616
The best way may be to link this to benchmark suites, so that the conditions are known and you can compute them once and store them for later use.
I.e., for benchmark 'OpenML100' you would report the average runtime, ACC,... over those 100 datasets.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading issue 616 and tracing how flows, benchmark suites, and evaluation metrics are represented. Define how an OpenML100-style suite would produce and retain average runtime, accuracy, and other metric values per flow; done means the aggregation conditions are known and the results can be reused.
Written by the indexing model from the issue text.
Assessment
- Domain
- analytics, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100