huggingface / huggingface/paperswithcode-feedback

[Dataset Request] Add Privy — PII detection in structured payloads

Open
#32 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
10
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Hi! I'd like to request indexing the **Privy** dataset and a leaderboard for **PII Detection** (structured / code-adjacent text).

### Dataset
- **HF:** [`piimb/privy`](https://huggingface.co/datasets/piimb/privy) — a duplicate of the authoritative [`beki/privy`](https://huggingface.co/datasets/beki/privy) (cite `beki/privy` for metadata)
- **Origin:** Benjamin Kilimnik (2022), part of the Pixie project — generator at https://github.com/pixie-io/pixie/tree/main/src/datagen/pii/privy ; write-up https://blog.px.dev/detect-pii/
- **Size:** ~308 MB (HF size category 100K–1M rows)
- **Distinctive:** PII embedded in **JSON / SQL / XML / HTML** payloads (not prose)
- **Labels:** 26 labels covering 60+ PII data providers (PERSON, CREDIT_CARD, US_SSN, IBAN_CODE, IP_ADDRESS, etc.); token/span labeled
- **Task:** PII Detection in structured/semi-structured text
- **Primary metric:** entity/span-level F1
- **License:** MIT

### Suggested results to seed the leaderboard
Verified from the model card:

| Model | Metric | F1 | Source |
|---|---|---|---|
| `beki/en_spacy_pii_distilbert` (spaCy transformer NER) | entity-level F (spaCy scorer) | **0.954** (P 0.953 / R 0.955) | [card](https://huggingface.co/beki/en_spacy_pii_distilbert) |
| `beki/flair-pii-distilbert` (Flair) | F1 | 0.9522 | [card](https://huggingface.co/beki/flair-pii-distilbert) |

⚠️ **Caveat:** both models were trained/evaluated on a **reduced 5-class** subset (PER, LOC, ORG, DATE_TIME, NRP), not the dataset's full 26-label set — so 0.954 is *not* full-PII-coverage F1. Note this when seeding the leaderboard. More entries can be imported from `pwc-archive/evaluation-tables` if present.

### Note
Fills a real gap — most PII benchmarks are prose; Privy tests detection inside structured payloads (SQL/JSON/XML/HTML). Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the Privy dataset metadata at beki/privy and the two linked model cards, then inspect this repository's existing dataset and leaderboard indexing entry points. Done means Privy is indexed for structured/semi-structured PII detection, with the requested span-level F1 leaderboard entries and an explicit note that the seeded scores use a reduced five-class evaluation.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface
Domain
data, machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.