microsoft / microsoft/PyRIT

FEAT Proposal: Turkish Prompt-Injection dataset loader (design/provenance discussion)

Open
#2,379 3 comments 0 reactions 1 assignee View on GitHub

@3nesdeniz is already working on this.

Since Sep 12, 2026.

Dominant language
Python
Stars
4.5k
Forks
893
Avg merge
3d 50m
Merged PRs (30d)
165

Description

Opening this as a **design-first proposal** before asking for a merge, following the catalog-direction guidance from @adrian-gavrila on #2175 (propose and get maintainer direction before building).

## What
A `SeedDatasetProvider` loader for the open **Turkish Prompt-Injection** dataset ([`3nesdeniz/turkish-prompt-injection-1k`](https://huggingface.co/datasets/3nesdeniz/turkish-prompt-injection-1k), CC BY 4.0): paired benign/attack Turkish examples across instruction override & system-prompt extraction, jailbreak personas, obfuscation & data exfiltration, and agentic tool abuse, with Turkish morphological variation.

## Why it could be valuable
- **Multilingual coverage gap:** PyRIT's injection-related seed datasets are English-first. Guardrails that pass in English often fail to generalize; Turkish (rich morphology) is a useful stress test for OWASP LLM01.
- **False-positive measurement:** the benign hard negatives share surface vocabulary with the attacks, so detector over-blocking is measurable, not just recall.

## Provenance (being upfront)
This is **not** backed by a peer-reviewed paper today. What exists: an openly published, CC BY 4.0 dataset with a `CITATION.cff`, a pinned immutable HF revision, and a documented deterministic/reproducible generation methodology; split-isolated train/val/test with no template leakage.

## Question for maintainers
Given that most loaders are paper-backed, **what evidence-of-value / provenance bar would you want here** before this is a candidate — e.g. a peer-reviewed or preprint write-up of the methodology, third-party usage, or something else? I'd rather align on direction first than repeat the sequencing issue from #2174/#2175.

A tested implementation is ready as a **draft** PR (#2365) purely for reference; I'm not requesting review of it until the direction here is settled.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.