dotnet / dotnet/machinelearning

Smart train memory handling for AutoML (ML.net 3)

Open
#6,925 1 comment 0 reactions 0 assignees View on GitHub
area-AutoML enhancement untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**Is your feature request related to a problem? Please describe.**

Let's see the current training settings:

var settings = new BinaryExperimentSettings
{
MaxExperimentTimeInSeconds = 30 * 60,
MaxModels = 10,
MaximumMemoryUsageInMegaByte = 7500,
};

ExperimentResult experimentResult = experiment
.Execute(trainDataView, nameof(MlModelRow.Label), nameof(MlModelRow.LearningGroup));

When the training takes more than 7500 Megabytes it will be canceled. But if we not set MaximumMemoryUsageInMegaByte this training take a lot of memory and in many case it will be more that our current pod memory ( > 36 Gb).

And during to logs it's often different amount of data. The very similar train set learning could set 10 Gb at first time and 30 Gb at second time.

**Describe the solution you'd like**
It will be perfect to have memory limitation as max memory ml.net can use for training without canceling. Like we have limitation for 7500 Mb and 1 training takes 2500 Mb so let's start 3 models training.

Contributor guide

Open the contributing guide

Research direction

The issue names BinaryExperimentSettings, MaximumMemoryUsageInMegaByte, and the experiment.Execute entry point; start by tracing how memory limits and concurrent model training are handled during AutoML experiments. Done means training can use a configured memory budget to choose safe parallelism without exceeding it or canceling unnecessarily, with behavior verified against the variable memory usage described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.