microsoft / microsoft/FLAML

BlendSearch: How do config constraints affect search performance?

Open
#977 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
4.4k
Forks
565
Avg merge
5d 8m
Merged PRs (30d)
17

Description

As I can read in the [docs](https://microsoft.github.io/FLAML/docs/Use-Cases/Tune-User-Defined-Function/#config_constraints-vs-metric_constraints), when passing a valid argument for the `config_constraints` parameter of `flaml.tune.run`, if a particular hyperparameter configuration violates any of the constraints imposed, its evaluation is prevented so as to avoid wasting processing time, which is great.

However, when looking in the [BlendSearch code](https://github.com/microsoft/FLAML/blob/f404b66a4c20d421c9cee465a6796ca76a0a8992/flaml/tune/searcher/blendsearch.py#L856) I can see that it assigns an infinite loss to that configuration.

So, from a theoretical standpoint, given that the global search is based on a regressor which has a smoothness prior (I think you mentioned BlendSearch using a Tree-structured Parzen Estimator, which, as per my limited understanding of the [paper](https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf) uses truncated Gaussian mixture models), wouldn't that cause configuration that are topologically adjacent to the edge of the feasible region to also be estimated to have very low performance and, thus, decrease the probability for them to be sampled?

I mean, I am aware that robust regressors should have some capacity to model kinks (discontinuities) in the surface they are estimating, but I wanted to ask how much of an extra cost would be incurred for the edge configurations to be taken into account, especially since it's possible that for some problems the optima have a high chance to lie right on the edge of feasibility.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.