Need for a sample or clarification on how to use PFI with AutoML in ML.NET
- Dominant language
- No language data
- Stars
- 4.8k
- Forks
- 6.1k
- Avg merge
- 15h 21m
- Merged PRs (30d)
- 370
Description
I've noticed that the users of [ML.NET](https://github.com/dotnet/machinelearning) continuously have had problems when using the `PermutationFeatureImportance` API with models created with `AutoML`. For example, we have these issues that are related to this problem:
https://github.com/dotnet/machinelearning/issues/5247
https://github.com/dotnet/machinelearning/issues/3972
https://github.com/dotnet/machinelearning/issues/4227
https://github.com/dotnet/machinelearning/issues/4196
https://github.com/dotnet/machinelearning/issues/3976
All of them are caused by the fact that users don't know how to "extract" the `linearPredictor` (needed for PFI) from a model retrieved from AutoML. Doing it would typically look like this: (as explained [here](https://github.com/dotnet/machinelearning/issues/5247#issuecomment-646232875)).
```C#
// Get BestRun from AutoML experiment:
RunDetail bestRun = experimentResult.BestRun;
// Extract the predictor:
var lastTransformer = ((TransformerChain )bestRun.Model).LastTransformer;
var linearPredictor = (ISingleFeaturePredictionTransformer)lastTransformer;
// Compute the permutation metrics for the linear model using the
// normalized data.
var permutationMetrics = mlContext.MulticlassClassification
.PermutationFeatureImportance(linearPredictor, transformedData,
permutationCount: 30);
```
What confuses users is the need for the casts of `(TransformerChain)` followed by `(ISingleFeaturePredictionTransformer)`... and it's understandable since, from my personal experience, the cast to `ISingleFeaturePredictionTransformer` is only used with PFI, and it's somewhat uncommon, so users are not aware of its existence or use (and I believe it's never mentioned in our docs); so users end up wondering if there's something wrong with their pipeline, with ML.NET, or if we even support using PFI with AutoML.
Aside of using these casts with AutoML, they might also be needed in other cases when working with PFI (such as after loading models from disk, or when calling PFI inside a Method that received the model as an `ITransformer`), but the opened issues mainly refer to AutoML.
I'm not sure if a whole new article would be needed, but I think that at least some mention to this would be useful in docs such as this one:
https://docs.microsoft.com/en-us/dotnet/machine-learning/how-to-guides/explain-machine-learning-model-permutation-feature-importance-ml-net#explain-the-model-with-permutation-feature-importance-pfi
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.