NVIDIA-NeMo / NVIDIA-NeMo/Anonymizer

feat: add domain-aware hyphen handling for partial-token detection (maybe after rewrite integration)

Open
#46 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement medium priority
Dominant language
Python
Stars
122
Forks
17
Avg merge
4d 2h
Merged PRs (30d)
11

Description

Priority Level

Medium (Nice to have)

Is your feature request related to a problem?

Summary

Handle hyphen-aware token boundaries (e.g. pre-K, African-American)
Suggestion: Add hyphen-aware token-boundary handling for entity detection once rewrite/domain context is available in the right layer.

Why

Today partial-token logic treats only letters, digits, and underscore as token-internal. That avoids code/query false positives, but it does not handle hyphenated forms well. We intentionally left hyphen logic out of PR #43 to avoid suppressing legitimate matches in normal prose (e.g. African-American)

Goal

Use domain-aware logic so hyphens can be treated as token boundaries differently depending on context:

  • In code-like or structured domains, treat - as token-internal when applying partial-token checks.
  • In normal prose, keep current behavior unless testing shows a safer rule.

Examples

  • Should likely drop in code-like text: internal-procID-id
  • Should likely preserve in prose: pre-Austin move
  • Should avoid awkward splits in hyphenated identity terms: African-American

Proposed follow-up

Update prompt guidance and postprocessing/token-boundary logic together so model instructions and regex behavior stay aligned.

Describe the solution you'd like

Moving domain classification to the top of the workflow for all modes; {if ...} block in the prompt.

Describe alternatives you've considered

No response

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no files or tests; start by reviewing PR #43 and tracing where domain classification, prompt guidance, and token-boundary postprocessing meet. Done means agreeing on context-sensitive hyphen behavior, preserving the listed prose examples, and keeping the prompt and postprocessing rules aligned.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.