dotnet / dotnet/machinelearning
Smart train memory handling for AutoML (ML.net 3)
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
**Is your feature request related to a problem? Please describe.**
Let's see the current training settings:
var settings = new BinaryExperimentSettings
{
MaxExperimentTimeInSeconds = 30 * 60,
MaxModels = 10,
MaximumMemoryUsageInMegaByte = 7500,
};
ExperimentResult experimentResult = experiment
.Execute(trainDataView, nameof(MlModelRow.Label), nameof(MlModelRow.LearningGroup));
When the training takes more than 7500 Megabytes it will be canceled. But if we not set MaximumMemoryUsageInMegaByte this training take a lot of memory and in many case it will be more that our current pod memory ( > 36 Gb).
And during to logs it's often different amount of data. The very similar train set learning could set 10 Gb at first time and 30 Gb at second time.
**Describe the solution you'd like**
It will be perfect to have memory limitation as max memory ml.net can use for training without canceling. Like we have limitation for 7500 Mb and 1 training takes 2500 Mb so let's start 3 models training.
Contributor guide
Research direction
The issue names BinaryExperimentSettings, MaximumMemoryUsageInMegaByte, and the experiment.Execute entry point; start by tracing how memory limits and concurrent model training are handled during AutoML experiments. Done means training can use a configured memory budget to choose safe parallelism without exceeding it or canceling unnecessarily, with behavior verified against the variable memory usage described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100