NVIDIA-NeMo / NVIDIA-NeMo/Anonymizer
feat: add domain-aware hyphen handling for partial-token detection (maybe after rewrite integration)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 122
- Forks
- 17
- Avg merge
- 4d 2h
- Merged PRs (30d)
- 11
Description
Priority Level
Medium (Nice to have)
Is your feature request related to a problem?
Summary
Handle hyphen-aware token boundaries (e.g. pre-K, African-American)
Suggestion: Add hyphen-aware token-boundary handling for entity detection once rewrite/domain context is available in the right layer.
Why
Today partial-token logic treats only letters, digits, and underscore as token-internal. That avoids code/query false positives, but it does not handle hyphenated forms well. We intentionally left hyphen logic out of PR #43 to avoid suppressing legitimate matches in normal prose (e.g. African-American)
Goal
Use domain-aware logic so hyphens can be treated as token boundaries differently depending on context:
- In code-like or structured domains, treat
-as token-internal when applying partial-token checks. - In normal prose, keep current behavior unless testing shows a safer rule.
Examples
- Should likely drop in code-like text:
internal-procID-id - Should likely preserve in prose:
pre-Austin move - Should avoid awkward splits in hyphenated identity terms:
African-American
Proposed follow-up
Update prompt guidance and postprocessing/token-boundary logic together so model instructions and regex behavior stay aligned.
Describe the solution you'd like
Moving domain classification to the top of the workflow for all modes; {if ...} block in the prompt.
Describe alternatives you've considered
No response
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no files or tests; start by reviewing PR #43 and tracing where domain classification, prompt guidance, and token-boundary postprocessing meet. Done means agreeing on context-sensitive hyphen behavior, preserving the listed prose examples, and keeping the prompt and postprocessing rules aligned.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100