huggingface / huggingface/paperswithcode-feedback

[Dataset Request] Add nvidia/Nemotron-PII — PII detection benchmark

Open
#31 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
10
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Hi! I'd like to request indexing the **Nemotron-PII** dataset and a leaderboard for **PII Detection**.

### Dataset
- **HF:** [`nvidia/Nemotron-PII`](https://huggingface.co/datasets/nvidia/Nemotron-PII)
- **Publisher:** NVIDIA
- **Size:** 200,000 rows (100K train / 100K test), synthetic, ~30 domains
- **Labels:** 50+ PII types; character-level spans (`{start, end, text, label}`) + inline-tagged text; English (US locale)
- **Task:** PII Detection / span extraction (token classification)
- **Primary metric:** token/span-level F1 (micro-averaged)
- **License:** CC-BY-4.0

### Suggested results to seed the leaderboard
⚠️ **Single-source, self-reported.** NVIDIA's dataset card publishes **no** baselines. The only public numbers I could find come from one community author (Kalyan KS), who trained the winning models — treat as a community claim, not an independent benchmark. Micro-F1 on 10K test instances:

| Model | Precision | Recall | F1 |
|---|---|---|---|
| GPT-4o-mini (zero-shot) | 95.21 | 67.05 | 78.69 |
| Ettin-32M (fine-tuned) | 95.96 | 95.49 | 95.73 |
| Ettin-68M (fine-tuned) | 96.35 | 96.19 | 96.27 |

Source: https://huggingface.co/blog/kalyan-ks/tiny-pii-entity-detection-models . Given the single-source provenance, it may be cleaner to **request the dataset/task first and leave results for the community/maintainer to populate** from `pwc-archive` or peer-reviewed work.

### Note
A large, recent, industry-domain PII dataset with fine-grained labels — good complement to AI4Privacy and Privy. Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by inspecting this repository's existing dataset and leaderboard indexing workflow, then review the nvidia/Nemotron-PII dataset card for its schema, labels, license, and evaluation task. Confirm how similar PII or token-classification benchmarks are represented, and leave results unseeded unless an accepted source is available; done means the dataset and PII Detection task are indexed with the stated metric and metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface
Domain
machine-learning, security
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.