Implement consistent cleaning for NatualLanguageFeaturizer
- Lingua principale
- Python
- Stelle
- 850
- Fork
- 96
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
In the `NaturalLanguageFeaturizer` component, cleaning of the data happens in the `_make_entityset` method. The LSA primitive is fit before the data cleaning happens, which I believe means that the custom LSA corpus is created using data that had not yet been cleaned.
However, when transform is called on the LSA component, this happens after `_make_entityset` has been called, and it appears that the cleaning of the dataframe happens in place. If I have followed things correctly, this would mean that LSA is fit using unclean data, but transform uses cleaned data to calculate the feature values.
This should be investigated further and if this is found to be an issue, consistent cleaning should be implemented to make sure that both fit and transform use the same clean data for computing LSA feature values.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.