huggingface / huggingface/paperswithcode-feedback
[Dataset Request] Add Privy — PII detection in structured payloads
- Dominant language
- No language data
- Stars
- 10
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Hi! I'd like to request indexing the **Privy** dataset and a leaderboard for **PII Detection** (structured / code-adjacent text).
### Dataset
- **HF:** [`piimb/privy`](https://huggingface.co/datasets/piimb/privy) — a duplicate of the authoritative [`beki/privy`](https://huggingface.co/datasets/beki/privy) (cite `beki/privy` for metadata)
- **Origin:** Benjamin Kilimnik (2022), part of the Pixie project — generator at https://github.com/pixie-io/pixie/tree/main/src/datagen/pii/privy ; write-up https://blog.px.dev/detect-pii/
- **Size:** ~308 MB (HF size category 100K–1M rows)
- **Distinctive:** PII embedded in **JSON / SQL / XML / HTML** payloads (not prose)
- **Labels:** 26 labels covering 60+ PII data providers (PERSON, CREDIT_CARD, US_SSN, IBAN_CODE, IP_ADDRESS, etc.); token/span labeled
- **Task:** PII Detection in structured/semi-structured text
- **Primary metric:** entity/span-level F1
- **License:** MIT
### Suggested results to seed the leaderboard
Verified from the model card:
| Model | Metric | F1 | Source |
|---|---|---|---|
| `beki/en_spacy_pii_distilbert` (spaCy transformer NER) | entity-level F (spaCy scorer) | **0.954** (P 0.953 / R 0.955) | [card](https://huggingface.co/beki/en_spacy_pii_distilbert) |
| `beki/flair-pii-distilbert` (Flair) | F1 | 0.9522 | [card](https://huggingface.co/beki/flair-pii-distilbert) |
⚠️ **Caveat:** both models were trained/evaluated on a **reduced 5-class** subset (PER, LOC, ORG, DATE_TIME, NRP), not the dataset's full 26-label set — so 0.954 is *not* full-PII-coverage F1. Note this when seeding the leaderboard. More entries can be imported from `pwc-archive/evaluation-tables` if present.
### Note
Fills a real gap — most PII benchmarks are prose; Privy tests detection inside structured payloads (SQL/JSON/XML/HTML). Thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the Privy dataset metadata at beki/privy and the two linked model cards, then inspect this repository's existing dataset and leaderboard indexing entry points. Done means Privy is indexed for structured/semi-structured PII detection, with the requested span-level F1 leaderboard entries and an explicit note that the seeded scores use a reduced five-class evaluation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100