dotnet / dotnet/machinelearning

AutoML doesn't allow ignored columns of incorrect type

Open
#5,573 0 comments 0 reactions 0 assignees View on GitHub
area-AutoML bug
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

When we are validating the datatypes of the incoming columns, we throw an error even if the column is marked as ignored.

For AutoML, any columns marked as ignored should skip this validation check. Users are currently blocked from handling the column featurization independently of AutoML in a [preFeaturizer](https://github.com/dotnet/machinelearning-samples/blob/0f01ae23a204d142923fb678c5f178b10592674d/samples/csharp/getting-started/AdvancedExperiment_AutoML/AdvancedTaxiFarePrediction/Program.cs#L99-L105) or for other purposes after AutoML completes.

### Error

Unhandled exception. System.ArgumentException: Only supported feature column types are Boolean, Single, and String. Please change the feature column unixTimeStamp of type Int64 to one of the supported types. (Parameter 'trainData')

Stack:
````
Unhandled exception. System.ArgumentException: Only supported feature column types are Boolean, Single, and String. Please change the feature column unixTimeStamp of type Int64 to one of the supported types. (Parameter 'trainData')
at Microsoft.ML.AutoML.UserInputValidationUtil.ValidateTrainData(IDataView trainData, ColumnInformation colum
nInformation)
at Microsoft.ML.AutoML.UserInputValidationUtil.ValidateExperimentExecuteArgs(IDataView trainData, ColumnInformation columnInformation, IDataView validationData, TaskKind task)
at Microsoft.ML.AutoML.ExperimentBase`2.ExecuteTrainValidate(IDataView trainData, ColumnInformation columnInfo, IDataView validationData, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
at Microsoft.ML.AutoML.ExperimentBase`2.Execute(IDataView trainData, IDataView validationData, ColumnInformation columnInformation, IEstimator`1 preFeaturizer, IProgress`1 progressHandler)
at ImgurClassifier.ConsoleApp.ModelBuilder.TrainAutoMLSubPipeline2(MLContext mlContext, IDataView trainDataView, IDataView validationDataView) in /Users/justinormont/Documents/src/ImgurClassifier/ImgurClassifier.ConsoleApp/ModelBuilder.cs:line 439
at ImgurClassifier.ConsoleApp.ModelBuilder.BuildTrainingPipelineUsingAutoML(MLContext mlContext, IDataView trainDataView, IDataView validationDataView, Boolean useAutoML) in /Users/justinormont/Documents/src/ImgurClassifier/ImgurClassifier.ConsoleApp/ModelBuilder.cs:line 159
at ImgurClassifier.ConsoleApp.ModelBuilder.CreateModel(MLContext mlContext) in /Users/justinormont/Documents/src/ImgurClassifier/ImgurClassifier.ConsoleApp/ModelBuilder.cs:line 65
at ImgurClassifier.ConsoleApp.Program.Main(String[] args) in /Users/justinormont/Documents/src/ImgurClassifier/ImgurClassifier.ConsoleApp/Program.cs:line 32
````
Location in code:
https://github.com/dotnet/machinelearning/blob/5dbfd8acac0bf798957eea122f1413209cdf07dc/src/Microsoft.ML.AutoML/Utils/UserInputValidationUtil.cs#L87-L100

### Repro
Add a column of type `long` to your loader class:
```c#
[ColumnName("unixTimeStamp"), LoadColumn(25)]
public long UnixTimeStamp { get; set; }
```

In the AutoML column information, set column as ignored: ([example code](https://github.com/dotnet/machinelearning-samples/blob/0f01ae23a204d142923fb678c5f178b10592674d/samples/csharp/getting-started/AdvancedExperiment_AutoML/AdvancedTaxiFarePrediction/Program.cs#L107-L110))
```c#
columnInformation.IgnoredColumnNames.Add("unixTimeStamp");
```

Load dataset in using AutoML, and receive the given error.

#### Work around

Mitigations:
* User can modify their loader function to either remove the column completely, though as the column is fully removed from the IDataView, this blocks them from working with the column in a [preFeaturizer](https://github.com/dotnet/machinelearning-samples/blob/0f01ae23a204d142923fb678c5f178b10592674d/samples/csharp/getting-started/AdvancedExperiment_AutoML/AdvancedTaxiFarePrediction/Program.cs#L99-L105) or for other ML․NET uses (e.g. having an [AutoML based stacked ensemble](https://github.com/justinormont/ImgurClassifier/blob/13b034fa311e0e314a74faf955066ca276296be7/ImgurClassifier.ConsoleApp/ModelBuilder.cs#L238-L240), and handling the specific column featurization in the main ML․NET pipeline).


* User can load the column as a `string` type and use the [ConvertType](https://docs.microsoft.com/en-us/dotnet/api/microsoft.ml.conversionsextensionscatalog.converttype?view=ml-dotnet) transform to convert to the needed type (e.g. `long`). AutoML will see the column as the acceptable `string` type, and still rightfully ignore the column.

### Fix
We should skip validation of any columns which are marked as `ColumnPurpose.Ignore`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.