bigscience-workshop / bigscience-workshop/pii_processing

add custom lexicon

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9
Forks
6
PR merge metrics
No merged PRs in 30d

Description

Add a custom lexicon to test_regex.py or even better to ontology_manager.py. Currently ontology_manager.py using a pattern match to find words potentially in a lexicon/ontology. You might want to have an exact dictionary to do a lexcion match, like "Barack Obama": "PUBLIC_FIGURE"}

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading test_regex.py and ontology_manager.py, especially the current pattern-matching path for lexicon or ontology terms. Trace how a detected term is classified, then define the custom dictionary behavior using the provided example and add coverage for exact matching. Done means custom entries are recognized with their configured labels without breaking existing matching.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
security
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.