dotnet / dotnet/machinelearning
LightGmb exception with an error code -1 and an error message of 'bad allocation'.
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
### System information
- **OS version/distro**: Windows 10 64 bit
- **.NET Version (eg., dotnet --info)**: .net core 3.0
### Issue
- **What did you do?**
Created a file with a label and a vector and trained against it.
- **What happened?**
Get an exception with an error code -1 and an error message of 'bad allocation'.
I can't find the error code in the source code so not sure what causes it. it happens after it releases about 30gb of memory. looks like right on
29.04.2019 20:20:01: [Source=**LightGBMMulticlass; Loading data for LightGBM**, Kind=Trace] Channel finished. Elapsed 01:52:28.7166971.
- **What did you expect?**
I'd expect this to work as it works with a smaller file.
[Crash.txt](https://github.com/dotnet/machinelearning/files/3128797/Crash.txt)
### Source code / logs
Loading the data
```
public static IDataView GetDataViewAsVector(MLContext mlContext, FileInfo trainingFile, long? maxRows = null)
{
var loader = mlContext.Data.CreateTextLoader(options: new TextLoader.Options()
{
Columns = new[] {
new TextLoader.Column(name:"Label", dataKind: DataKind.String, index: 0),
new TextLoader.Column(name:"Features",dataKind:DataKind.Single,minIndex:1,maxIndex:40731)
},
HasHeader = false,
Separators = new[] { '|' },
UseThreads = true,
MaxRows=maxRows
});
var dv = loader.Load(trainingFile.FullName);
return dv;
}
```
Making the model
```
var dataView = DataViewUtils.GetDataViewAsVector(mlContext, trainingFile, null);
dataset = mlContext.Data.TrainTestSplit(dataView, testFraction: 0.1);
var start = DateTime.Now;
(long? training, long? validating) = ((dataset.TrainSet.GetColumn(Label).LongCount(), dataset.TestSet.GetColumn(Label).LongCount()));
this.OnUpdate?.Invoke($"Start training LightGBM on {training.Value:N0} rows of data");
this.OnUpdate?.Invoke($"Testing LightGBM wil be on {validating.Value:N0} rows of data");
var options =new LightGbmMulticlassTrainer.Options {
LabelColumnName = KeyColumn,
FeatureColumnName = Features,
Silent= false,
Verbose=true,
NumberOfThreads=8
};
var pipeline = mlContext.Transforms.Conversion.MapValueToKey(outputColumnName: KeyColumn, inputColumnName: Label)
// .Append(mlContext.Transforms.Concatenate(outputColumnName: Features, inputColumnNames: Mapper.GetFieldNames()))
.Append(mlContext.MulticlassClassification.Trainers.LightGbm(options))
.Append(mlContext.Transforms.CopyColumns(inputColumnName: KeyColumn, outputColumnName: nameof(PredictedResult.PredictedLabelIndex)))
;
// Train the model.
var model = pipeline.Fit(dataset.TrainSet);
```
Contributor guide
Assessment
This issue has not been assessed yet.