GSA / GSA/os3_taxonomy_constructor
Define Product Description Cleaning Heuristics
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
**Problem Description**
We need to be sure that the product descriptions we pass to a Search API are quality, i.e. returning accurate results.
**Proposed Solution**
We need to define some basic heuristics that can be programmatically applied to *every* product description. We need to be careful in this process to not include text cleaning fixes that are overfit to certain product categories as there could be unintended consequences within other, more minority, classes.
**Describe alternatives you've considered**
On my first pass, I stripped all numbers and punctuation from the product descriptions before lowercasing, tokenizing and lemmatizing. Then, if a Search API query returned 0 results, I'd strip non-dictionary words from the string (using `enchant`) and try the query again so long as the resultant string wasn't empty.
**Additional context**
We need to keep in mind that the solution will need to generalize as much as possible to other product categories.
Contributor guide
Assessment
This issue has not been assessed yet.