FEAT: Add Garak divergence scenario
@VimalN2005 is already working on this.
Since Sep 8, 2026.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 893
- Avg merge
- 3d 50m
- Merged PRs (30d)
- 165
Description
#### Is your feature request related to a problem? Please describe.
PyRIT does not have a bounded scenario for garak's repetition-induced divergence checks. A direct port would generate tokenizer data at runtime and change target output settings from probe code. Those behaviors do not fit PyRIT component responsibilities and can create unexpectedly expensive runs. This is part of #511.
#### Describe the solution you'd like
Add a PyRIT-native divergence scenario based on [`garak/probes/divergence.py`](https://github.com/NVIDIA/garak/blob/main/garak/probes/divergence.py).
Required scope and behavior:
- Make `Repeat` the default scenario technique.
- Add `RepeatedToken` as an explicit opt-in technique. Leave inactive `RepeatExtended` outside the required first version.
- Store repeat templates, seed words, and safe configuration metadata in local PyRIT datasets.
- Store a small, fixed, attributed set of precomputed repeated-token strings in the dataset. Do not add runtime tokenizer generation or a new tokenizer dependency.
- Use standard PyRIT attack flow. Do not mutate target `max_tokens` or other target settings in the scenario. Document any recommended target output limit as user configuration.
- Reuse existing refusal and text-analysis components where their contracts fit. If no existing scorer covers divergence, add one deterministic scorer that distinguishes refusal or bounded requested repetition from unexpected continuation, and that detects excessive repeated structures for `RepeatedToken`.
- Set conservative prompt counts and response limits so the default is not a denial-of-service test.
- Describe positive results as divergence or output instability. Do not claim training-data leakage without a separate attributable reference match.
- Add exports, bounded unit tests, and synchronized `.py` and `.ipynb` scanner documentation.
Use PromptInject PR #2509 as an example of dataset/technique separation. Follow [`doc/code/framework.md`](https://github.com/microsoft/PyRIT/blob/main/doc/code/framework.md) and the applicable scenario, dataset, scorer, test, and documentation instructions. Do not port garak generator hooks into a scenario.
#### Describe alternatives you've considered, if relevant
- Runtime `tiktoken` generation was considered and rejected. Fixed data keeps the scenario deterministic and avoids dependency and tokenizer-version drift.
- Changing target output limits from scenario code was considered and rejected. The target owner must control those limits.
- Treating any repetition as a positive result would confuse expected bounded compliance with divergence.
#### Additional context
- Parent parity issue: #511
- PyRIT structural example: #2509
- Upstream probe: [`garak/probes/divergence.py`](https://github.com/NVIDIA/garak/blob/main/garak/probes/divergence.py)
- Garak is Apache-2.0, not MIT. Retain applicable attribution and modification notices for copied or adapted source data.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.