dotnet / dotnet/machinelearning
Default parameters not taking into account in GenerateCandidateConfigurations
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
Per my understanding:
1) Default values for trainers are set based on experience from a wide range of dataset. Therefore, they should be a good starting point.
2) AutoML start by running with default values and then randomly 10 experiments around it.
3) The random values can be used to estimate direction for next parameters search.
However, it seems GenerateCandidateConfigurations does not receive parameters for the 1st run with default parameters in previousRun attribute (SmacSweeper.cs). It likely means forestPredictor and GredyPlusRandomSearch do not know what are the defaults and what was the metrics for the first run, so it can only rely on random experiments. This may have detrimental effect on the next suggested parameters because there have been no known good combinations of parameters in previousRuns.
I believe the default parameters should be added to history and then previousRuns variable, so that SmacSweeper can use them to propose new parameters.
If my understanding is correct it may also affect ModelBuilder, although it probably does not use SmacSweeper.
Contributor guide
Assessment
This issue has not been assessed yet.