automl / automl/DeepCAVE

New plugin: Hardness of AutoML problem

Open
#29 0 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Python
Stars
104
Forks
15
PR merge metrics
No merged PRs in 30d

Description

Hi,

We often wonder how hard an AutoML problem is. Can we therefore add some metrics regarding that?
For example
* an eCDF plot for the cost distribution (i.e., a hard AutoML task should have only a few very well-performing configurations)
* uni-modal metric from the automl loss landscape paper (but evaluated on our surrogate models)
* convexity metric from the automl loss landscape paper (but evaluated on our surrogate models)

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are identified. Start by locating the surrogate-model analysis and visualization components, then clarify the proposed cost-distribution, uni-modal, and convexity metrics and how completion would be evaluated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-visualization, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.