microsoft / microsoft/FLAML

tune: support for incremental searches

Open
#817 11 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
4.4k
Forks
565
Avg merge
5d 8m
Merged PRs (30d)
17

Description

So, let's assume you get a new dataset every month and need to train a new model on that data as soon as you receive it. You don't care if the model doesn't perform very well on old data, you care about the newest data only. There's enough of a concept drift within that data, that the optimal model hyperparameter configuration differs a bit from month to month so it helps to do a little tuning every time you get data. Accumulated over several months, however, the pairwise performance differences among a set of configurations might end up being significant. Of course, there are certain stable regions in the design space which have always yielded, e.g., terrible results and should mostly be avoided in the future, too.

Now, how might one implement such an iterative scenario using FLAML tune?

### Option 1: searching the best config within a constrained sub-space of the original design space
I'm thinking that the dataset can be represented by just another integer parameter in the design space, i.e., the index of the current month [0, 1, 2, ..., 12]. However, this parameter shouldn't be sampled by the algorithm, but its value should be specified manually each month. Also unlike regular parameters, we're not looking for a global optimum any more, but for the local optimum, specific only for that particular month.

As such, we'd need a mechanism to have the optimizer sample a fixed value during a single optimization session. I think Optuna has something for this called [`PartialFixedSampler`](https://optuna.readthedocs.io/en/stable/reference/samplers/generated/optuna.samplers.PartialFixedSampler.html) (not exactly sure, though, if this is exactly right).

Alternatively, when launching the search for the current month's optimum, in the design space, we could change the bounds of the month index variable (e.g., `qrandint(11, 11, 1)` if we want to select month 11). However, results from previous months, e.g., from month 10, will, basically, be considered invalid. Will it still be possible to warm start this month's optimization using previous months' trials through, `tune.run`'s `points_to_evaluate` and `evalulated_rewards` arguments, or will the optimizer complain that it received an invalid configuration because 10 is not in [11, 11]? Also, will this incremental approach change the surface learned by the BO, i.e., will it have to unlearn/relearn some changing regions of the design space, and thus lose performance?

### Option 2: instances / transfer learning
I can see other optimizer packages address this issue of multiple datasets with dedicated mechanism. E.g., SMAC3 has [instances](https://automl.github.io/SMAC3/main/advanced_usage/4_instances.html), while OpenBox calls this "[transfer learning](https://open-box.readthedocs.io/en/latest/advanced_usage/transfer_learning.html)". SMAC3 can also assign some values to so-called "instance features", and I believe these can better inject the local smoothness prior along the temporal dimension, and that might have an advantage over just treating all months as independent data sets. Maybe you guys also have something similar, but I missed it or wasn't documented or something.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.