dotnet / dotnet/machinelearning

Passing serialized TensorFlow Example to TF Serving SavedModel

Open
#5,336 14 comments 0 reactions 0 assignees View on GitHub
area-TensorFlow enhancement
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

### System information

- **OS version/distro**: macOS 10.15
- **.NET Version (eg., dotnet --info)**: 3.1.301

### Issue

- **What did you do?**
I would like to use the PredictionEnginePool (eventually) in combination with a pretrained Tensorflow Model that I exported using the [Estimator.export_saved_model](https://www.tensorflow.org/guide/saved_model?hl=en#savedmodels_from_estimators) function in combination with [`build_parsing_serving_input_receiver_fn`](https://www.tensorflow.org/api_docs/python/tf/estimator/export/build_parsing_serving_input_receiver_fn).

Specifically, I went through this tutorial: https://www.tensorflow.org/tfx/tutorials/transform/census. Below, you can find the Tensorflow Serving signature definition according to `saved_model_cli`.

- **What happened?**
The `input_example_tensor` input expects a serialized [Example](https://github.com/tensorflow/tensorflow/blob/r2.3/tensorflow/core/example/example.proto) message (a binary buffer, not a text string). This does not work using the ML.NET library because it [re-encodes the data](https://github.com/dotnet/machinelearning/blob/release/1.5.1/src/Microsoft.ML.TensorFlow/TensorflowTransform.cs#L837) that I'm providing as the model input.
- **What did you expect?**
There should be the option in ML.NET to pass raw binary data as a TFString to the model (maybe as a `byte[]` or `ReadOnlyMemory`?).

### Source code / logs
Saved model signature:
```bash
$ saved_model_cli show --dir ./my_saved_model --tag_set serve --signature_def serving_default
The given SavedModel SignatureDef contains the following input(s):
inputs['inputs'] tensor_info:
dtype: DT_STRING
shape: (-1)
name: input_example_tensor:0
The given SavedModel SignatureDef contains the following output(s):
outputs['classes'] tensor_info:
dtype: DT_STRING
shape: (-1, 2)
name: head/Tile:0
outputs['scores'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 2)
name: head/predictions/probabilities:0
Method name is: tensorflow/serving/classify
```

My code:
```csharp
class ModelInput {
[ColumnName("input_example_tensor"), VectorType(1)]
public string[] InputExampleTensor { get; set; }
}
class ModelPrediction {
[ColumnName("head/Tile:0"), VectorType(2)]
public string[] Classes { get; set; }

[ColumnName("head/predictions/probabilities:0"), VectorType(2)]
public float[] Prediction { get; set; }
}

var mlContext = new MLContext();

var pipeline = mlContext.Model.LoadTensorFlowModel("my_saved_model")
.ScoreTensorFlowModel(
outputColumnNames: new[] { "head/Tile:0", "head/predictions/probabilities:0" },
inputColumnNames: new[] { "input_example_tensor" }
);

// Train the model
// Since we are simply using a pre-trained TensorFlow model,
// we can "train" it against an empty dataset
var emptyTrainingSet = mlContext.Data.LoadFromEnumerable(new List());
var mlModel = pipeline.Fit(emptyTrainingSet);
var engine = mlContext.Model.CreatePredictionEngine(mlModel);

// Example is a Protobuf-Class, generated from example.proto
var example = new Example();
// filling the example with features omitted

var input = new ModelInput {
InputExampleTensor = new[] { new string(example.ToByteArray().Select(x => (char)x).ToArray()) }
};

var prediction = engine.Predict(input);
```

Which fails with:
```
W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at example_parsing_ops.cc:92 : Invalid argument: Could not parse example input, value: ''
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.