dotnet / dotnet/machinelearning

LightGmb exception with an error code -1 and an error message of 'bad allocation'.

Open
#3,615 4 comments 0 reactions 0 assignees View on GitHub
area-Trees Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

### System information

- **OS version/distro**: Windows 10 64 bit
- **.NET Version (eg., dotnet --info)**: .net core 3.0

### Issue

- **What did you do?**
Created a file with a label and a vector and trained against it.
- **What happened?**
Get an exception with an error code -1 and an error message of 'bad allocation'.
I can't find the error code in the source code so not sure what causes it. it happens after it releases about 30gb of memory. looks like right on
29.04.2019 20:20:01: [Source=**LightGBMMulticlass; Loading data for LightGBM**, Kind=Trace] Channel finished. Elapsed 01:52:28.7166971.

- **What did you expect?**
I'd expect this to work as it works with a smaller file.

[Crash.txt](https://github.com/dotnet/machinelearning/files/3128797/Crash.txt)

### Source code / logs
Loading the data
```
public static IDataView GetDataViewAsVector(MLContext mlContext, FileInfo trainingFile, long? maxRows = null)
{
var loader = mlContext.Data.CreateTextLoader(options: new TextLoader.Options()
{
Columns = new[] {
new TextLoader.Column(name:"Label", dataKind: DataKind.String, index: 0),
new TextLoader.Column(name:"Features",dataKind:DataKind.Single,minIndex:1,maxIndex:40731)
},
HasHeader = false,
Separators = new[] { '|' },
UseThreads = true,
MaxRows=maxRows
});
var dv = loader.Load(trainingFile.FullName);
return dv;
}
```
Making the model
```
var dataView = DataViewUtils.GetDataViewAsVector(mlContext, trainingFile, null);
dataset = mlContext.Data.TrainTestSplit(dataView, testFraction: 0.1);

var start = DateTime.Now;
(long? training, long? validating) = ((dataset.TrainSet.GetColumn(Label).LongCount(), dataset.TestSet.GetColumn(Label).LongCount()));

this.OnUpdate?.Invoke($"Start training LightGBM on {training.Value:N0} rows of data");
this.OnUpdate?.Invoke($"Testing LightGBM wil be on {validating.Value:N0} rows of data");

var options =new LightGbmMulticlassTrainer.Options {
LabelColumnName = KeyColumn,
FeatureColumnName = Features,
Silent= false,
Verbose=true,
NumberOfThreads=8
};

var pipeline = mlContext.Transforms.Conversion.MapValueToKey(outputColumnName: KeyColumn, inputColumnName: Label)
// .Append(mlContext.Transforms.Concatenate(outputColumnName: Features, inputColumnNames: Mapper.GetFieldNames()))
.Append(mlContext.MulticlassClassification.Trainers.LightGbm(options))
.Append(mlContext.Transforms.CopyColumns(inputColumnName: KeyColumn, outputColumnName: nameof(PredictedResult.PredictedLabelIndex)))
;

// Train the model.
var model = pipeline.Fit(dataset.TrainSet);
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.