dotnet / dotnet/machinelearning

Provide a way to append\concatentate multiple IDataViews

Open
#4,005 0 comments 5 reactions 0 assignees View on GitHub
area-Core enhancement Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

### System information

- ML.NET - 1.2.0:

### Issue

There should be a way to append or concatenate multiple IDataViews together.

Here's the scenario:
The [new ranking sample](https://github.com/dotnet/machinelearning-samples/pull/549) needs the ability to train the model using two datasets that are each loaded from a separate text file and have the same schema - specifically, there is a (1) Training dataset and (2) Validation dataset, that need to be combined. For example, refer to step #3 in the steps outlined below which the sample is based on.

Here's the steps shown in the sample - generally, the pattern to train, validate, and test a model includes the following steps:
1. The model is trained on the **training** dataset. The model's metrics are then evaluated using the **validation** dataset.
2. Step #1 is repeated by retraining and reevaluating the model until the desired metrics are achieved. The outcome of this step is a pipeline that applies the necessary data transformations and trainer.
3. The pipeline is used to train on the combined **training** + **validation** datasets. The model's metrics are then evaluated on the **testing** dataset (exactly once) -- this is the final set of metrics used to measure the model's quality.
4. The final step is to retrain the pipeline on **all** of the combined **training** + **validation** + **testing** datasets. This model is then ready to be deployed into production.

Today to achieve this, the sample has to first load the data from a text file, then create an enumerable so that the datasets can be concatenated - this process would be greatly simplified if you could append/concatenate two IDataViews together:

```

//Load training data (has a header)
IDataView trainData = mlContext.Data.LoadFromTextFile(TrainDatasetPath, separatorChar: '\t', hasHeader: true);

//Load validation data (has a header)
IDataView validationData = mlContext.Data.LoadFromTextFile(ValidationDatasetPath, separatorChar: '\t', hasHeader: false);

// Combine the training and validation datasets.
var validationDataEnum = mlContext.Data.CreateEnumerable(validationData, false);
var trainDataEnum = mlContext.Data.CreateEnumerable(trainData, false);
var trainValidationDataEnum = validationDataEnum.Concat(trainDataEnum);
IDataView trainValidationData = mlContext.Data.LoadFromEnumerable(trainValidationDataEnum);
```

NOTE: I also considered creating a text loader to load multiple text files (as described [here])(https://docs.microsoft.com/en-us/dotnet/api/microsoft.ml.data.textloader.load?view=ml-dotnet#Microsoft_ML_Data_TextLoader_Load_Microsoft_ML_Data_IMultiStreamSource_); however, one of the data files included a header while the other didn't. It looks like to create a TextLoader for multiple files, that the file headers must be consistent across files.

### Source code / logs

Note that there is a method today that provides the ability to append rows - we should consider exposing this publicly:

https://github.com/dotnet/machinelearning/blob/70ef7ecd43b031b481a4047ea361da5e2f360336/src/Microsoft.ML.Data/DataView/AppendRowsDataView.cs#L23-L31

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.