dotnet / dotnet/machinelearning

Let ML.NET cancel unmanaged code.

Open
#6,465 14 comments 0 reactions 0 assignees View on GitHub
area-AutoML enhancement
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**System Information:**
- OS & Version: Windows 11 [Version 10.0.22621.819]
- ML.NET Version: ML.NET v2.0.0
- .NET Version: .NET 7.0.100

**Describe the bug**
I run an ML.NET experiment for 10 hours. I limited the training for FastTree and FastForest algorithms and I made sure that every new best model is saved even if the whole run fails. I locked Windows and let it run during the night.
When I checked it in the morning (12 hours after the experiment started) I couldn't even log in, I had to do a power off-power on to get access. When I checked the logs it looked like the last experiment was saved after ~5 hours of running.
The dataset wasn't particularly big this time, it was only a 205 MB CSV file. My system has 32 GB RAM, and I set the maximum memory usage for the experiment to 20 GB.

**To Reproduce**
Steps to reproduce the behavior:
1. Create a pipeline that includes only FastTree and FastForest, for example:
```
var pipeline =
mlContext.Auto().Featurizer(trainTestData.TrainSet, numericColumns: new[] { "Features" })
.Append(mlContext.Auto().Regression(useFastTree: true, useLbfgs: false, useSdca: false, useFastForest: true, useLgbm: false));

```
2. Create an ML.NET experiment and set its memory limit. I set it to 20 GB, because I usually have ~24 GB available when I start the training.
```
var experiment = mlContext.Auto().CreateExperiment();

experiment
.SetPipeline(pipeline)
.SetTrainingTimeInSeconds(trainingTimeInSeconds)
.SetRegressionMetric(RegressionMetric.RSquared, labelColumn: "Label")
.SetDataset(trainTestData.TrainSet, trainTestData.TestSet)
.SetMonitor(monitor)
.SetMaximumMemoryUsageInMegaByte(20 * 1024);
```
4. Run the experiment for at least 10 hours
5. Check if you can still use the system after 5+ hours of running

**Expected behavior**
I would expect the experiment to run without a system-freezing halt.

**Screenshots, Code, Sample Projects**
The 205 MB CSV dataset can be downloaded from here:
[BBD_20221122__TrainingData.Sleep.MLP12__MLP12_0p25Hz-250Hz__Session_SegmentedData_Sleep_Level__15782rows.zip](https://drive.google.com/file/d/1FIE9Qqmt44WEmqFpZESjrIbL-sEafigX/view?usp=sharing)

**Additional context**
My project is open-source so both the source and the data are available. Let me know if you need them for testing.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.