dotnet / dotnet/machinelearning
Cannot use a deserialized FastTree model for feature contribution
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
**System Information (please complete the following information):**
- OS & Version: [e.g. Windows 10] Windows 10 build 19043
- ML.NET Version: [e.g. ML.NET v1.5.5] 1.5.5
- .NET Version: [e.g. .NET 5.0] 5.0.104
**Describe the bug**
A clear and concise description of what the bug is.
While attempting to determine feature contributions for a single sample using a deserialized FastTree model, I found that I could not cast the model's LastTransformer to ISingleFeaturePredictionTransformer in order to use it as a parameter for MLContext.Transforms.CalculateFeatureContribution. The type of LastTransformer at compile time is ITransform. Upon inspection at runtime, however, its type is Microsoft.ML.IPredictorProducing, and this type appears to have members that can be used as parameters for CalculateFeatureContribution. Hoever, because IPredictionPreducing is defined as internal, it is not possible to cast LastTransformer to this type at compile time.
It is possible, however, to get feature contributions using the model object resulting from the Fit call. This is not practical in a production environment because we cannot train a model every time we need to make a prediction, since the model needs to predict on a large number of samples very quickly. Is there a way to serialize the model such that it can be loaded by an application and used to get feature contribution data for each sample it predicts?
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
See code below.
**Expected behavior**
A clear and concise description of what you expected to happen.
I expected to be able to cast the LastTransformer property to something that can be used to calculate the feature contributions, but this is not possible with a loaded FastTree model.
**Screenshots, Code, Sample Projects**
If applicable, add screenshots, code snippets, or sample projects to help explain your problem.
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Data;
using Microsoft.ML.Trainers.FastTree;
using Microsoft.ML.Calibrators;
namespace ftpredict
{
class Program
{
private class FeatureData
{
[LoadColumn(0), ColumnName("ID")]
public string SHA { get; set; }
[LoadColumn(1), ColumnName("Label")]
public string Label { get; set; }
[LoadColumn(2), ColumnName("InFeatures")]
public float[] InFeatures { get; set; }
}
private class Prediction
{
public float Score { get; set; }
public float Probability { get; set; }
public float[] FeatureContributions { get; set; }
}
static void Main(string[] args)
{
Predictor p = new Predictor();
p.LoadModel(args[0]);
IEnumerable features = p.ReadSingleTestItemFromFile(args[1]);
if (features != null)
{
Prediction pred = p.Predict(features);
Console.WriteLine("Confidence = {0}", pred.Probability);
}
}
class Predictor
{
private PredictionEngine Engine;
private SchemaDefinition schemaDefinition;
MLContext mlContext;
ITransformer Model;
public void LoadModel(string modelPath)
{
using (var modelFileStream = File.OpenRead(modelPath))
{
try
{
const int FEATUREARRAYINDEX = 2;
mlContext = new MLContext();
Model = mlContext.Model.Load(modelFileStream, out DataViewSchema schema);
Microsoft.ML.Data.VectorDataViewType featureVectorView = (Microsoft.ML.Data.VectorDataViewType)schema[FEATUREARRAYINDEX].Type;
schemaDefinition = SchemaDefinition.Create(typeof(FeatureData));
schemaDefinition["InFeatures"].ColumnType = new VectorDataViewType(NumberDataViewType.Single, featureVectorView.Size);
Engine = mlContext.Model.CreatePredictionEngine(Model,
inputSchemaDefinition: schemaDefinition);
}
catch (FormatException e)
{
throw new Exception($"Could not load classifier from file: {e.Message}");
}
finally
{
modelFileStream.Close();
}
}
}
public IEnumerable ReadSingleTestItemFromFile(string testDataPath)
{
IEnumerable testData = null;
using (var testDataStream = new System.IO.StreamReader(testDataPath))
{
try
{
string line = testDataStream.ReadLine();
string[] strData = line.Split(',');
testData = strData.Select(a => float.Parse(a, System.Globalization.CultureInfo.InvariantCulture)).ToList();
}
catch (Exception e)
{
throw new Exception($"Failed loading test data from file: {e.Message}");
}
testDataStream.Close();
}
return testData;
}
public Prediction Predict(IEnumerable features)
{
float confidence;
FeatureData featureData = new FeatureData
{
InFeatures = features.Select(a => a).ToArray()
};
var transformation = Engine.Predict(featureData);
confidence = transformation.Probability;
IEnumerable fd = new[] { featureData };
var data = mlContext.Data.LoadFromEnumerable(fd, schemaDefinition);
var transformedData = Model.Transform(data);
// Based on this example, post by Antoniovs1029: https://github.com/dotnet/machinelearning/issues/4937
// Cast fails (lastTransformer1 is null)
var lastTransformer1 = (Model as TransformerChain).LastTransformer as ISingleFeaturePredictionTransformer;
// Based on this example, post by Antoniovs1029: https://github.com/dotnet/machinelearning/issues/4937
// Cast fails, lastTransformer2 is null
var lastTransformer2 = (Model as TransformerChain).LastTransformer as BinaryPredictionTransformer>;
// At compile time, this is an ITransformer, so .Model.SubModel cannot be referenced
var lastTransformer3 = (Model as TransformerChain).LastTransformer;
// At runtime, lastTransformer's type is Microsoft.ML.Data.BinaryPredictionTransformer>
// but can't be used at compile time because IPredictorProducing is internal
//var lastTransformer4 = lastTransformer as Microsoft.ML.Data.BinaryPredictionTransformer>;
// This line causes an exception because casting lastTransformer1 to
// ISingleFeaturePredictionTransformer fails
var featureContributionCalculation = mlContext.Transforms.CalculateFeatureContribution(lastTransformer1, normalize: false);
var featureContributionData = featureContributionCalculation.Fit(transformedData).Transform(transformedData);
var shuffledSubset = mlContext.Data.TakeRows(mlContext.Data.ShuffleRows(featureContributionData), 10);
var scoringEnumerator = mlContext.Data.CreateEnumerable(shuffledSubset, true);
IEnumerable featureContributions = scoringEnumerator.Select(a => a.FeatureContributions);
return transformation;
}
}
}
}
**Additional context**
Add any other context about the problem here.
Contributor guide
Assessment
This issue has not been assessed yet.