dotnet / dotnet/machinelearning

TextFeaturizingEstimator.Options KeepNumbers and KeepPunctuations not exportable to ONNX?

Open
#6,317 1 comment 1 reaction 0 assignees View on GitHub
area-ONNX bug
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

**System Information (please complete the following information):**
- OS & Version: Windows 10
- .NET Version: .NET 6

```xml

```

**Describe the bug**

As seen here: https://github.com/jonathanpeppers/inclusive-code-reviews-ml/pull/29#discussion_r944879120

A pipeline such as:

```csharp
var dataProcessPipeline = mlContext.Transforms.Conversion.MapValueToKey("isnegative", "isnegative")
.Append(mlContext.Transforms.Text.FeaturizeText("text_tf", new TextFeaturizingEstimator.Options
{
//NOTE: not exportable to ONNX
KeepNumbers = false,
KeepPunctuations = false,
// NOTE: these work
KeepDiacritics = true,
CaseMode = TextNormalizingEstimator.CaseMode.Lower,
}, "text"))
```

Hits an exception such as:

```
Unhandled exception. System.Collections.Generic.KeyNotFoundException: The given key 'text_TextNormalizer' was not present in the dictionary.
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
at Microsoft.ML.Model.OnnxConverter.OnnxContextImpl.GetVariableName(String colName)
at Microsoft.ML.Transforms.Text.WordTokenizingTransformer.Mapper.SaveAsOnnx(OnnxContext ctx)
at Microsoft.ML.Data.RowToRowMapperTransform.Microsoft.ML.Model.OnnxConverter.ISaveAsOnnx.SaveAsOnnx(OnnxContext ctx)
at Microsoft.ML.Model.OnnxConverter.SaveOnnxCommand.ConvertTransformListToOnnxModel(OnnxContextImpl ctx, IChannel ch, IDataView inputData, IDataView outputData, LinkedList`1 transforms, HashSet`1 inputColumnNamesToDrop, HashSet`1 outputColumnNamesToDrop)
at Microsoft.ML.OnnxExportExtensions.ConvertToOnnxProtobufCore(IHostEnvironment env, OnnxContextImpl ctx, ITransformer transform, IDataView inputData, String[] outputColumnNamesToKeep)
at Microsoft.ML.OnnxExportExtensions.ConvertToOnnxProtobuf(ModelOperationsCatalog catalog, ITransformer transfor at Microsoft.ML.OnnxExportExtensions.ConvertToOnnx(ModelOperationsCatalog catalog, ITransformer transform, IDataView inputData, Stream stream)
at InclusiveCodeReviews.ConsoleApp.ModelBuilder.SaveModel(MLContext mlContext, IDataView dataView, ITransformer
mlModel, String modelRelativePath, DataViewSchema modelInputSchema) in C:\src\inclusive-code-reviews-ml\ml.net\InclusiveCodeReviews.ConsoleApp\ModelBuilder.cs:line 102
at InclusiveCodeReviews.ConsoleApp.ModelBuilder.CreateModel() in C:\src\inclusive-code-reviews-ml\ml.net\InclusiveCodeReviews.ConsoleApp\ModelBuilder.cs:line 49
at Program.$(String[] args) in C:\src\inclusive-code-reviews-ml\ml.net\InclusiveCodeReviews.Convert\Program.cs:line 4
```

**To Reproduce**

Steps to reproduce the behavior:
1. Run this project:

https://github.com/jonathanpeppers/inclusive-code-reviews-ml/tree/main/ml.net/InclusiveCodeReviews.Convert

2. Uncomment these two lines:

https://github.com/jonathanpeppers/inclusive-code-reviews-ml/blob/486f7737174702233825ceddf28adb5cc7912f43/ml.net/InclusiveCodeReviews.ConsoleApp/ModelBuilder.cs#L59-L61

**Expected behavior**

In particular, we want to use `KeepPunctuations=false` and export to ONNX.

**Screenshots, Code, Sample Projects**

See above.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.