automl / automl/SMAC3

[Refactor] Move cost validation to `tell`

Open
#950 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.2k
Forks
245
Avg merge
5d 8h
Merged PRs (30d)
5

Description

Currently the `runner` validates the costs. This is all well and good until having to use the `ask` and `tell` interface. Reporting a correct `TrialValue` requires the user to re-implement all of the validation logic found within the runner here:

https://github.com/automl/SMAC3/blob/731854e8effe72d76cb07636458e783143f32dc5/smac/runner/target_function_runner.py#L156-L215

This validation logic makes sure that the costs are ordered correctly if reported as a `dict` and also ensures the `crash_cost` is extended to the length of the `cost` list if using multi-objective and `crash_cost` is a single float. Re-implementing this just to use `ask` and `tell` is a little verbose.

This issue is best illustrated by trying to create a more thorough example of using `ask-and-tell` which also uses multi-objective and reports a `TrialValue` with failed results, where `Scenario` did not explicitly provide a `Scenario(crash_cost=...)` for each objective.

https://automl.github.io/SMAC3/main/examples/1_basics/3_ask_and_tell.html#ask-and-tell

I'm not sure how best to approach this but one possibility might be to add the validation logic to `runhistory.add`, but this is just a guess and may not be the best solution.

Contributor guide

Open the contributing guide

Research direction

Start with smac/runner/target_function_runner.py at the linked validation block, then read the ask-and-tell example and the TrialValue and runhistory.add entry points. Compare validation used by runner with what tell users must currently reproduce. Done means ask-and-tell supports ordered dict costs and extends a scalar crash_cost for multi-objective failed results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.