dotnet / dotnet/machinelearning
Smart train memory handling for AutoML (ML.net 3)
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
**Is your feature request related to a problem? Please describe.**
Let's see the current training settings:
var settings = new BinaryExperimentSettings
{
MaxExperimentTimeInSeconds = 30 * 60,
MaxModels = 10,
MaximumMemoryUsageInMegaByte = 7500,
};
ExperimentResult experimentResult = experiment
.Execute(trainDataView, nameof(MlModelRow.Label), nameof(MlModelRow.LearningGroup));
When the training takes more than 7500 Megabytes it will be canceled. But if we not set MaximumMemoryUsageInMegaByte this training take a lot of memory and in many case it will be more that our current pod memory ( > 36 Gb).
And during to logs it's often different amount of data. The very similar train set learning could set 10 Gb at first time and 30 Gb at second time.
**Describe the solution you'd like**
It will be perfect to have memory limitation as max memory ml.net can use for training without canceling. Like we have limitation for 7500 Mb and 1 training takes 2500 Mb so let's start 3 models training.
Contributor guide
Assessment
This issue has not been assessed yet.