bigscience-workshop / bigscience-workshop/pii_processing
add custom lexicon
- Dominant language
- Python
- Stars
- 9
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
Add a custom lexicon to test_regex.py or even better to ontology_manager.py. Currently ontology_manager.py using a pattern match to find words potentially in a lexicon/ontology. You might want to have an exact dictionary to do a lexcion match, like "Barack Obama": "PUBLIC_FIGURE"}
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading test_regex.py and ontology_manager.py, especially the current pattern-matching path for lexicon or ontology terms. Trace how a detected term is classified, then define the custom dictionary behavior using the provided example and add coverage for exact matching. Done means custom entries are recognized with their configured labels without breaking existing matching.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- security
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100