dotnet / dotnet/machinelearning
Passing serialized TensorFlow Example to TF Serving SavedModel
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
### System information
- **OS version/distro**: macOS 10.15
- **.NET Version (eg., dotnet --info)**: 3.1.301
### Issue
- **What did you do?**
I would like to use the PredictionEnginePool (eventually) in combination with a pretrained Tensorflow Model that I exported using the [Estimator.export_saved_model](https://www.tensorflow.org/guide/saved_model?hl=en#savedmodels_from_estimators) function in combination with [`build_parsing_serving_input_receiver_fn`](https://www.tensorflow.org/api_docs/python/tf/estimator/export/build_parsing_serving_input_receiver_fn).
Specifically, I went through this tutorial: https://www.tensorflow.org/tfx/tutorials/transform/census. Below, you can find the Tensorflow Serving signature definition according to `saved_model_cli`.
- **What happened?**
The `input_example_tensor` input expects a serialized [Example](https://github.com/tensorflow/tensorflow/blob/r2.3/tensorflow/core/example/example.proto) message (a binary buffer, not a text string). This does not work using the ML.NET library because it [re-encodes the data](https://github.com/dotnet/machinelearning/blob/release/1.5.1/src/Microsoft.ML.TensorFlow/TensorflowTransform.cs#L837) that I'm providing as the model input.
- **What did you expect?**
There should be the option in ML.NET to pass raw binary data as a TFString to the model (maybe as a `byte[]` or `ReadOnlyMemory`?).
### Source code / logs
Saved model signature:
```bash
$ saved_model_cli show --dir ./my_saved_model --tag_set serve --signature_def serving_default
The given SavedModel SignatureDef contains the following input(s):
inputs['inputs'] tensor_info:
dtype: DT_STRING
shape: (-1)
name: input_example_tensor:0
The given SavedModel SignatureDef contains the following output(s):
outputs['classes'] tensor_info:
dtype: DT_STRING
shape: (-1, 2)
name: head/Tile:0
outputs['scores'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 2)
name: head/predictions/probabilities:0
Method name is: tensorflow/serving/classify
```
My code:
```csharp
class ModelInput {
[ColumnName("input_example_tensor"), VectorType(1)]
public string[] InputExampleTensor { get; set; }
}
class ModelPrediction {
[ColumnName("head/Tile:0"), VectorType(2)]
public string[] Classes { get; set; }
[ColumnName("head/predictions/probabilities:0"), VectorType(2)]
public float[] Prediction { get; set; }
}
var mlContext = new MLContext();
var pipeline = mlContext.Model.LoadTensorFlowModel("my_saved_model")
.ScoreTensorFlowModel(
outputColumnNames: new[] { "head/Tile:0", "head/predictions/probabilities:0" },
inputColumnNames: new[] { "input_example_tensor" }
);
// Train the model
// Since we are simply using a pre-trained TensorFlow model,
// we can "train" it against an empty dataset
var emptyTrainingSet = mlContext.Data.LoadFromEnumerable(new List());
var mlModel = pipeline.Fit(emptyTrainingSet);
var engine = mlContext.Model.CreatePredictionEngine(mlModel);
// Example is a Protobuf-Class, generated from example.proto
var example = new Example();
// filling the example with features omitted
var input = new ModelInput {
InputExampleTensor = new[] { new string(example.ToByteArray().Select(x => (char)x).ToArray()) }
};
var prediction = engine.Predict(input);
```
Which fails with:
```
W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at example_parsing_ops.cc:92 : Invalid argument: Could not parse example input, value: ''
```
Contributor guide
Assessment
This issue has not been assessed yet.