dotnet / dotnet/machinelearning

Fix Recursive Parsing for PFI

Open
#6,084 17 comments 1 reaction 0 assignees View on GitHub
area-Core bug Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**System Information (please complete the following information):**
- Windows 11
- ML.NET Version 1.7, AutoML 0.19.0
- .NET Version: .NET Core 3.1

**Describe the bug**
ArgumentNullException while attempting to retrieve PFI
The model provided does not have a compatible predictor (Parameter 'lastTransformer')

Stack trace
at Microsoft.ML.Runtime.Contracts.CheckValue[T](IExceptionContext ctx, T val, String paramName, String msg)
at Microsoft.ML.PermutationFeatureImportanceExtensions.PermutationFeatureImportance[TMetric,TResult](IHostEnvironment env, ITransformer model, IDataView data, Func`1 resultInitializer, Func`2 evaluationFunc, Func`3 deltaFunc, Int32 permutationCount, Boolean useFeatureWeightFilter, Nullable`1 numberOfExamplesToUse)
at Microsoft.ML.PermutationFeatureImportanceExtensions.PermutationFeatureImportance(RegressionCatalog catalog, ITransformer model, IDataView data, String labelColumnName, Boolean useFeatureWeightFilter, Nullable`1 numberOfExamplesToUse, Int32 permutationCount)

**To Reproduce**

Not sure how best to create a minimal reproducible example including data, but here's the core of what I did (based on https://github.com/dotnet/machinelearning/pull/5934)

```
MLContext mlContext = new MLContext();

ColumnInferenceResults columnInference = mlContext.Auto().InferColumns(trainDataPath, "Label", groupColumns: false);

TextLoader textLoader = mlContext.Data.CreateTextLoader(columnInference.TextLoaderOptions);
IDataView trainDataView = textLoader.Load(trainDataPath);
IDataView testDataView = textLoader.Load(testDataPath);

IEstimator preFeaturizer =
mlContext.Transforms.Categorical.OneHotEncoding(
new T().GetOneHotInputColumnNames().Select(_ => new InputOutputColumnPair(_)).ToArray()
);

ColumnInformation columnInformation = columnInference.ColumnInformation;
columnInformation.IgnoredColumnNames.AddIfMissing("Foo");
columnInformation.CategoricalColumnNames.Remove("Foo");
columnInformation.NumericColumnNames.Remove("Foo");

BinaryExperimentSettings experimentSettings = new BinaryExperimentSettings()
{
MaxExperimentTimeInSeconds = experimentTime,
OptimizingMetric = BinaryClassificationMetric.F1Score
};

var experiment = mlContext.Auto().CreateBinaryClassificationExperiment(experimentSettings);

ExperimentResult experimentResult = experiment.Execute(trainDataView, columnInformation, preFeaturizer, progress);

RunDetail bestRun = experimentResult.BestRun;

// Exception thrown here
var permutationFeatureImportance = mlContext
.Regression
.PermutationFeatureImportance(bestRun.Model, testDataViewWithBestScore, permutationCount: 3);
```
The `LastTransformer` of `bestRun.Model` is `BinaryPredictionTransformer>`

**Expected behavior**
Retrieve the PFI information.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.