GSA / GSA/os3_taxonomy_constructor

Define Product Description Cleaning Heuristics

Open
#4 2 comments 0 reactions 1 assignee Claimed by @csmcallister View on GitHub
good first issue nlp
Dominant language
Python
Stars
1
Forks
2
PR merge metrics
No merged PRs in 30d

Description

**Problem Description**
We need to be sure that the product descriptions we pass to a Search API are quality, i.e. returning accurate results.

**Proposed Solution**
We need to define some basic heuristics that can be programmatically applied to *every* product description. We need to be careful in this process to not include text cleaning fixes that are overfit to certain product categories as there could be unintended consequences within other, more minority, classes.

**Describe alternatives you've considered**
On my first pass, I stripped all numbers and punctuation from the product descriptions before lowercasing, tokenizing and lemmatizing. Then, if a Search API query returned 0 results, I'd strip non-dictionary words from the string (using `enchant`) and try the query again so long as the resultant string wasn't empty.

**Additional context**
We need to keep in mind that the solution will need to generalize as much as possible to other product categories.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.