[Refactor] Move cost validation to `tell`
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 245
- Avg merge
- 5d 8h
- Merged PRs (30d)
- 5
Description
Currently the `runner` validates the costs. This is all well and good until having to use the `ask` and `tell` interface. Reporting a correct `TrialValue` requires the user to re-implement all of the validation logic found within the runner here:
https://github.com/automl/SMAC3/blob/731854e8effe72d76cb07636458e783143f32dc5/smac/runner/target_function_runner.py#L156-L215
This validation logic makes sure that the costs are ordered correctly if reported as a `dict` and also ensures the `crash_cost` is extended to the length of the `cost` list if using multi-objective and `crash_cost` is a single float. Re-implementing this just to use `ask` and `tell` is a little verbose.
This issue is best illustrated by trying to create a more thorough example of using `ask-and-tell` which also uses multi-objective and reports a `TrialValue` with failed results, where `Scenario` did not explicitly provide a `Scenario(crash_cost=...)` for each objective.
https://automl.github.io/SMAC3/main/examples/1_basics/3_ask_and_tell.html#ask-and-tell
I'm not sure how best to approach this but one possibility might be to add the validation logic to `runhistory.add`, but this is just a guess and may not be the best solution.
Contributor guide
Research direction
Start with smac/runner/target_function_runner.py at the linked validation block, then read the ask-and-tell example and the TrialValue and runhistory.add entry points. Compare validation used by runner with what tell users must currently reproduce. Done means ask-and-tell supports ordered dict costs and extends a scalar crash_cost for multi-objective failed results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100