dotnet / dotnet/machinelearning

Discrepancy in NgramExtractorTrasform, NgramExtractingTransformer and NgramExtractingEstimator.

Open
#2,895 2 comments 0 reactions 0 assignees View on GitHub
area-Transforms Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

If you search for `NgramExtract` in the solution, the following three main classes pop up.

1. NgramExtractorTransform (in WordBagTransform.cs)
2. NgramExtractingTransformer (in NgramTransform.cs)
3. NgramExtractingEstimator (in NgramTrasnform.cs)

`2` and `3` seem to be the actual classes where ngram extraction logic is written. However, `1` uses `2` and `3` with a pre-processing step where if input is text it is first converted to terms using ValueToKeyMappingTransformer.

First, `NgramExtractorTransform` does not seem to be in correct file i.e filename and class name do not match.
Second, the `NgramExtractorTransform` is not doing ngram extraction instead composing two different estimators (NgramExtractingEstimator and ValueToKeyMappingEstimator).

I think `NgramExtractorTransform` be renamed to `WordBagTransform` or something appropirate.

CC: @Ivanidzo4ka, @TomFinley, @sfilipi, @rogancarr.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.