dotnet / dotnet/machinelearning

Smart train memory handling for AutoML (ML.net 3)

Open
#6,925 1 comment 0 reactions 0 assignees View on GitHub
area-AutoML enhancement untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**Is your feature request related to a problem? Please describe.**

Let's see the current training settings:

var settings = new BinaryExperimentSettings
{
MaxExperimentTimeInSeconds = 30 * 60,
MaxModels = 10,
MaximumMemoryUsageInMegaByte = 7500,
};

ExperimentResult experimentResult = experiment
.Execute(trainDataView, nameof(MlModelRow.Label), nameof(MlModelRow.LearningGroup));

When the training takes more than 7500 Megabytes it will be canceled. But if we not set MaximumMemoryUsageInMegaByte this training take a lot of memory and in many case it will be more that our current pod memory ( > 36 Gb).

And during to logs it's often different amount of data. The very similar train set learning could set 10 Gb at first time and 30 Gb at second time.

**Describe the solution you'd like**
It will be perfect to have memory limitation as max memory ml.net can use for training without canceling. Like we have limitation for 7500 Mb and 1 training takes 2500 Mb so let's start 3 models training.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.