bigscience-workshop / bigscience-workshop/data_tooling
Improve lexicon and ontology_manager.py
- Dominant language
- HTML
- Stars
- 91
- Forks
- 47
- PR merge metrics
- No merged PRs in 30d
Description
Currently ontology_manager.py using a pattern match to find words potentially in a lexicon/ontology. You might want to have an exact dictionary to do a lexcion match, like "Barack Obama": "PUBLIC_FIGURE". Also, we need to incorporate better stop words and language specific stopwords to prevent the infering a word belongs in the lexicon when it's a stop words (like "maybe", "must", etc.)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading ontology_manager.py and trace the current pattern-based lexicon or ontology matching. Determine where exact dictionary matching and stop-word filtering belong; done means matching supports entries such as "Barack Obama": "PUBLIC_FIGURE" and excludes general and language-specific stop words such as "maybe" and "must".
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100