microsoft / microsoft/PyRIT

FEAT Add Agentic Prompt-Injection Boundary Pairs dataset loader

Open
#2,174 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.5k
Forks
893
Avg merge
3d 50m
Merged PRs (30d)
165

Description

#### Is your feature request related to a problem? Please describe.

PyRIT does not currently expose a paired prompt-injection dataset designed to test the boundary between legitimate agent workflows and adversarial instructions. Attack-only collections are useful for recall, but they do not let users build or inspect false-positive-aware test surfaces where benign and attack prompts intentionally share the same scenario and vocabulary.

#### Describe the solution you'd like

Add a remote loader for the CC BY 4.0 [Agentic Prompt-Injection Boundary Pairs](https://huggingface.co/datasets/3nesdeniz/agentic-prompt-injection-boundary-pairs) dataset.

The proposed loader would:

- load the published train, validation, test, or combined surface;
- default to attack examples for red-team use while allowing benign or paired loading;
- preserve `pair_id`, `scenario_id`, `attack_family`, `pair_family`, `source_context`, `risk_domain`, `target_boundary`, `split`, and label metadata;
- support typed filters for label, split, attack family, and source context;
- keep scenario-isolated split semantics intact;
- include mocked unit tests, documentation, and bibliography updates.

The dataset contains 1,200 English records organized as 600 controlled benign/attack pairs across 50 enterprise and agentic scenarios and 12 attack families. JSONL and Parquet artifacts, deterministic build scripts, validation, checksums, and a 30% stratified manual-review record are public.

#### Describe alternatives you've considered, if relevant

Users can download the JSONL files directly, but that bypasses PyRIT dataset discovery, metadata preservation, filtering, and consistent `SeedDataset` handling. A generic file loader would also lose the paired-boundary semantics unless every consumer rebuilt the same mapping.

#### Additional context

Dataset: https://huggingface.co/datasets/3nesdeniz/agentic-prompt-injection-boundary-pairs

Source and build pipeline: https://github.com/3nesdeniz/agentic-prompt-injection-boundary-pairs

I maintain the dataset and can keep the loader schema synchronized with tagged dataset releases.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting PyRIT's existing dataset loaders and SeedDataset handling, then trace how metadata and filters are represented. Implement the requested train, validation, test, combined, attack, benign, and paired loading behavior with pair metadata preserved. Add mocked unit tests, documentation, and bibliography updates; verify scenario-isolated splits and typed filters.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, data, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.