dotnet / dotnet/docs

Need for a sample or clarification on how to use PFI with AutoML in ML.NET

Open
#19,006 7 comments 0 reactions 1 assignee Claimed by @luisquintanilla View on GitHub
dotnet-ml/svc
Dominant language
No language data
Stars
4.8k
Forks
6.1k
Avg merge
15h 21m
Merged PRs (30d)
370

Description

I've noticed that the users of [ML.NET](https://github.com/dotnet/machinelearning) continuously have had problems when using the `PermutationFeatureImportance` API with models created with `AutoML`. For example, we have these issues that are related to this problem:

https://github.com/dotnet/machinelearning/issues/5247
https://github.com/dotnet/machinelearning/issues/3972
https://github.com/dotnet/machinelearning/issues/4227
https://github.com/dotnet/machinelearning/issues/4196
https://github.com/dotnet/machinelearning/issues/3976

All of them are caused by the fact that users don't know how to "extract" the `linearPredictor` (needed for PFI) from a model retrieved from AutoML. Doing it would typically look like this: (as explained [here](https://github.com/dotnet/machinelearning/issues/5247#issuecomment-646232875)).

```C#
// Get BestRun from AutoML experiment:
RunDetail bestRun = experimentResult.BestRun;

// Extract the predictor:
var lastTransformer = ((TransformerChain )bestRun.Model).LastTransformer;
var linearPredictor = (ISingleFeaturePredictionTransformer)lastTransformer;

// Compute the permutation metrics for the linear model using the
// normalized data.
var permutationMetrics = mlContext.MulticlassClassification
.PermutationFeatureImportance(linearPredictor, transformedData,
permutationCount: 30);
```

What confuses users is the need for the casts of `(TransformerChain)` followed by `(ISingleFeaturePredictionTransformer)`... and it's understandable since, from my personal experience, the cast to `ISingleFeaturePredictionTransformer` is only used with PFI, and it's somewhat uncommon, so users are not aware of its existence or use (and I believe it's never mentioned in our docs); so users end up wondering if there's something wrong with their pipeline, with ML.NET, or if we even support using PFI with AutoML.

Aside of using these casts with AutoML, they might also be needed in other cases when working with PFI (such as after loading models from disk, or when calling PFI inside a Method that received the model as an `ITransformer`), but the opened issues mainly refer to AutoML.

I'm not sure if a whole new article would be needed, but I think that at least some mention to this would be useful in docs such as this one:
https://docs.microsoft.com/en-us/dotnet/machine-learning/how-to-guides/explain-machine-learning-model-permutation-feature-importance-ml-net#explain-the-model-with-permutation-feature-importance-pfi

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.