dotnet / dotnet/machinelearning

LightGMB bad allocation crash

Open
#6,817 0 comments 0 reactions 0 assignees View on GitHub
area-Trees untriaged
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

- OS & Version: Windows 10
- ML.NET Version: ML.NET v2.0.1 & AutoML v0.20.1
- .NET Version: .NET 7.0
- 16gb ram
- 4 cores

Hello,
during an AutoML regression training session, after more than three hours of training and successful training of 5 models `[LightGBM] [Warning] bad allocation` warnings shows in the console, and after a while the program crashes.

### About my dataset:
111 275 columns and 872 rows (including label column). Contains only floats from -1 to 1.

### Code responsible for training the model:
```cs
public async Task> TrainModel(DataOperationsCatalog.TrainTestData trainValidationData)
{
Logger.Log($"Running the experiment...");

AutoMLExperiment experiment = MLContext.Auto().CreateExperiment();

experiment
.SetPipeline(pipeline)
.SetRegressionMetric(RegressionMetric.MeanAbsoluteError)
.SetTrainingTimeInSeconds(maxTrainingTime)
.SetDataset(trainValidationData);

CancellationTokenSource cts = new();

AutoMLMonitor monitor = new(pipeline, maxTrainingTime, maxTrainingIterations, cts);
experiment.SetMonitor(monitor);

await experiment.RunAsync(cts.Token);
return monitor.GetCompletedTrials();
}
```

### My pipeline is simple:
```cs
pipeline = MLContext.Auto().Regression(useFastForest: false, useFastTree: false, useLbfgs: false, useLgbm: true, useSdca: false);
```

### Logs:
(In the timestamp, the first number is the application running time (hh:mm) and the second number is the local time (hh:mm:ss))
```
[0:00 - 06:59:59] Loading data set...
[0:01 - 07:01:09] Creating data view...
[0:01 - 07:01:11] Running the experiment...
[0:01 - 07:01:11] Model 1 started training using LightGbmRegression algorithm
[0:01 - 07:01:20] 10s/80000s (0.01 %) - Model finished training in 9.44s using LightGbmRegression algorithm (CPU: 95.51 %, RAM: 1828.41) - result: 15.08 %
[0:01 - 07:01:20] ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ New best result! (15.08 %) ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲
[0:01 - 07:01:20] Model 2 started training using LightGbmRegression algorithm
[0:03 - 07:02:42] 91s/80000s (0.11 %) - Model finished training in 81.75s using LightGbmRegression algorithm (CPU: 102.15 %, RAM: 1971.35) - result: 8.46 %
[0:03 - 07:02:42] ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ New best result! (8.46 %) ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲
[0:03 - 07:02:42] Model 3 started training using LightGbmRegression algorithm
[0:10 - 07:09:56] 526s/80000s (0.66 %) - Model finished training in 434.11s using LightGbmRegression algorithm (CPU: 101.56 %, RAM: 2008.12) - result: 6.49 %
[0:10 - 07:09:56] ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ New best result! (6.49 %) ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲
[0:10 - 07:09:56] Model 4 started training using LightGbmRegression algorithm
[3:09 - 10:08:58] 11267s/80000s (14.08 %) - Model finished training in 10741.15s using LightGbmRegression algorithm (CPU: 104.30 %, RAM: 5345.96) - result: 5.58 %
[3:09 - 10:08:58] ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲ New best result! (5.58 %) ▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲▲
[3:09 - 10:08:58] Model 5 started training using LightGbmRegression algorithm
[3:21 - 10:20:58] 11987s/80000s (14.98 %) - Model finished training in 720.38s using LightGbmRegression algorithm (CPU: 102.34 %, RAM: 2526.46) - result: 8.69 %
[3:21 - 10:20:58] Model 6 started training using LightGbmRegression algorithm
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] [LightGBM] [Warning] bad allocation
bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
[LightGBM] [Warning] bad allocation
(...) more of those
```
[full log.txt](https://github.com/dotnet/machinelearning/files/12567896/full.log.txt)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.