dotnet / dotnet/machinelearning

Default parameters not taking into account in GenerateCandidateConfigurations

Open
#6,025 1 comment 0 reactions 0 assignees View on GitHub
area-AutoML enhancement
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

Per my understanding:
1) Default values for trainers are set based on experience from a wide range of dataset. Therefore, they should be a good starting point.
2) AutoML start by running with default values and then randomly 10 experiments around it.
3) The random values can be used to estimate direction for next parameters search.

However, it seems GenerateCandidateConfigurations does not receive parameters for the 1st run with default parameters in previousRun attribute (SmacSweeper.cs). It likely means forestPredictor and GredyPlusRandomSearch do not know what are the defaults and what was the metrics for the first run, so it can only rely on random experiments. This may have detrimental effect on the next suggested parameters because there have been no known good combinations of parameters in previousRuns.

I believe the default parameters should be added to history and then previousRuns variable, so that SmacSweeper can use them to propose new parameters.

If my understanding is correct it may also affect ModelBuilder, although it probably does not use SmacSweeper.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.