bigscience-workshop / bigscience-workshop/pii_processing
improve stopwords in ontology_manager.py
Open
- Dominant language
- Python
- Stars
- 9
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Improve the stopwords list in ontology_manager.py for example using https://github.com/rspeer/wordfreq.
This allows us to do better lexicon. For example, words like "best" and "much" are stopwords and should not be returned as NER/PII entities.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.