FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 893
- Avg merge
- 3d 50m
- Merged PRs (30d)
- 165
Description
#### Is your feature request related to a problem? Please describe.
PyRIT does not currently expose a paired prompt-injection dataset designed to test the boundary between legitimate agent workflows and adversarial instructions. Attack-only collections are useful for recall, but they do not let users build or inspect false-positive-aware test surfaces where benign and attack prompts intentionally share the same scenario and vocabulary.
#### Describe the solution you'd like
Add a remote loader for the CC BY 4.0 [Agentic Prompt-Injection Boundary Pairs](https://huggingface.co/datasets/3nesdeniz/agentic-prompt-injection-boundary-pairs) dataset.
The proposed loader would:
- load the published train, validation, test, or combined surface;
- default to attack examples for red-team use while allowing benign or paired loading;
- preserve `pair_id`, `scenario_id`, `attack_family`, `pair_family`, `source_context`, `risk_domain`, `target_boundary`, `split`, and label metadata;
- support typed filters for label, split, attack family, and source context;
- keep scenario-isolated split semantics intact;
- include mocked unit tests, documentation, and bibliography updates.
The dataset contains 1,200 English records organized as 600 controlled benign/attack pairs across 50 enterprise and agentic scenarios and 12 attack families. JSONL and Parquet artifacts, deterministic build scripts, validation, checksums, and a 30% stratified manual-review record are public.
#### Describe alternatives you've considered, if relevant
Users can download the JSONL files directly, but that bypasses PyRIT dataset discovery, metadata preservation, filtering, and consistent `SeedDataset` handling. A generic file loader would also lose the paired-boundary semantics unless every consumer rebuilt the same mapping.
#### Additional context
Dataset: https://huggingface.co/datasets/3nesdeniz/agentic-prompt-injection-boundary-pairs
Source and build pipeline: https://github.com/3nesdeniz/agentic-prompt-injection-boundary-pairs
I maintain the dataset and can keep the loader schema synchronized with tagged dataset releases.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting PyRIT's existing dataset loaders and SeedDataset handling, then trace how metadata and filters are represented. Implement the requested train, validation, test, combined, attack, benign, and paired loading behavior with pair metadata preserved. Add mocked unit tests, documentation, and bibliography updates; verify scenario-isolated splits and typed filters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, data, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100