dotnet / dotnet/machinelearning

Error message 'Dataset row count exceeded the maximum count of 2146435071' in Fit() method

Open
#5,751 0 comments 0 reactions 0 assignees View on GitHub
area-Trees enhancement Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

### System information

- **Windows10 Home**:
- **.NET Version 2.1.0**:
- **ML.NET Version 1.5.5**:

### Issue

- **We are currently dealing with large sizes of data.
Is it possible to include a record size larger than INTMAX in the size of training data?**

![exception_row_count](https://user-images.githubusercontent.com/13389351/115135773-0a937680-a056-11eb-82c2-df471f964815.png)

### Source code / logs

public static void CreateModel1()
{
// Load Data
IDataView trainingDataView = mlContext.Data.LoadFromTextFile(
path: USDJPY_CSV_FILEPATH,
hasHeader: true,
separatorChar: ',',
allowQuoting: true,
allowSparse: false);

// Build training pipeline
IEstimator trainingPipeline = BuildTrainingPipeline(mlContext);

// Evaluate quality of Model
Evaluate(mlContext, trainingDataView, trainingPipeline);

// Train Model
ITransformer mlModel = TrainModel(mlContext, trainingDataView, trainingPipeline);

// Save model
SaveModel(mlContext, mlModel, MODEL_FILEPATH, trainingDataView.Schema);

}
public static IEstimator BuildTrainingPipeline(MLContext mlContext)
{
// Data process configuration with pipeline data transformations
var dataProcessPipeline = mlContext.Transforms.Categorical.OneHotHashEncoding(new[] { new InputOutputColumnPair("period", "period") })
.Append(mlContext.Transforms.Concatenate("Features", new[] { "period", "st_price", "volume" }));

// Set the training algorithm
var trainer = mlContext.Regression.Trainers.LightGbm(labelColumnName: "ed_price", featureColumnName: "Features");
var trainingPipeline = dataProcessPipeline.Append(trainer);

return trainingPipeline;
}

public static ITransformer TrainModel(MLContext mlContext, IDataView trainingDataView, IEstimator trainingPipeline)
{
Console.WriteLine("=============== Training model ===============");

ITransformer model = trainingPipeline.Fit(trainingDataView);

Console.WriteLine("=============== End of training process ===============");
return model;
}

private static void Evaluate(MLContext mlContext, IDataView trainingDataView, IEstimator trainingPipeline)
{
// Cross-Validate with single dataset (since we don't have two datasets, one for training and for evaluate)
// in order to evaluate and get the model's accuracy metrics
Console.WriteLine("=============== Cross-validating to get model's accuracy metrics ===============");
//var crossValidationResults = mlContext.Regression.CrossValidate(trainingDataView, trainingPipeline, numberOfFolds: 5, labelColumnName: "ed_price");
//PrintRegressionFoldsAverageMetrics(crossValidationResults);
}

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.