dotnet / dotnet/machinelearning
Improving column purpose detection for sparse datasets
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
AutoML does poorly on a few text datasets. For example, a text dataset we benchmark on has an accuracy of 0.60 vs. an expected accuracy of 0.85.
This is caused by us detecting the text columns columns as **categorical** instead of **free text**. For the this dataset, this is due to the text column being 84% blank (a sparsely filled out column).
**To fix:**
We need to detect the column purpose only on the set (non-blank) values.
Recommend subtracting the blank values from `data.Count`:
https://github.com/dotnet/machinelearning/blob/227da9d7db2ce80b073cc64bfd067b04e6189de1/src/Microsoft.ML.AutoML/ColumnInference/PurposeInference.cs#L148-L158
Currently `avgLength`, `cardinalityRatio`, `avgSpaces` are artificially lower due to the missing values.
Contributor guide
Assessment
This issue has not been assessed yet.