FEAT: Add ANSI escape output scorers and Garak AnsiEscape scenario
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 893
- Avg merge
- 3d 50m
- Merged PRs (30d)
- 165
Description
### Part of #511
`AnsiAttackConverter` (#577 → #597) covers the *prompt* side of garak's `ansiescape` probe. The other half discussed on #577 — a scorer that checks whether the model actually emitted terminal control sequences — was the first item under "Next Steps" in #597 but was never followed up. There is also no scenario for garak's two active probes, `ansiescape.AnsiRaw` (tier `OF_CONCERN`) and `ansiescape.AnsiEscaped` (`INFORMATIONAL`).
Right now PyRIT can send ANSI prompts but can't tell whether the target produced something a terminal, log viewer or renderer downstream would act on.
#### Proposed scope
Two PRs, scorers first since they're useful on their own:
1. **Scorers** in `pyrit/score/true_false/regex/`, following `XSSOutputScorer`:
- `AnsiEscapeOutputScorer` — raw sequences: `ESC [` (CSI), `ESC ]` (OSC) and the C1 code points U+009B / U+009D. Parity with garak's `detectors.ansiescape.Raw`.
- `EscapedAnsiOutputScorer` — escaped text forms that go live once something unescapes them: `\x1b`, `\033`, `\u001b`, `\27`, `\e` followed by `[` or `]`, plus `\x9b` / `\x9d`. Parity with `detectors.ansiescape.Escaped`.
Two classes rather than one with a mode, so raw and escaped findings stay distinguishable in memory and metrics — same split garak makes with its tiers.
2. **Scenario** `garak.AnsiEscape` with `Raw` / `Escaped` techniques on `PromptSendingAttack`, prompts from a local `garak_ansi_escape` dataset (garak's asks × tasks, repeat stubs, unescape stubs), mirroring `WebInjection` / `Doctor`. Plus exports, tests, the synchronized scanner docs and `third_party/garak-provenance.json` entries.
`AnsiAttackConverter` and the Foundry `ansi_attack` technique stay as they are.
#### One upstream detail
garak's `RAW_COMMANDS` contains `"\27"`, which in a Python string literal is octal U+0017, not ESC, so that entry can never match what it's meant to. I'd leave it out of the raw scorer. The decimal form is still covered on the escaped side, where `\27[` is literal text.
#### One question before I start
Is the two-PR split fine, or would you rather see scorers and scenario together the way #2064 shipped `PolicyPuppetryConverter` with `Doctor`?
#### References
- Parent: #511; converter: #577, #597
- Upstream (pinned to the garak revision in `third_party/garak-provenance.json`): [`probes/ansiescape.py`](https://github.com/NVIDIA/garak/blob/3f50ea5ff9cd7050099940647c15c39b07a93392/garak/probes/ansiescape.py), [`detectors/ansiescape.py`](https://github.com/NVIDIA/garak/blob/3f50ea5ff9cd7050099940647c15c39b07a93392/garak/detectors/ansiescape.py), [`resources/ansi.py`](https://github.com/NVIDIA/garak/blob/3f50ea5ff9cd7050099940647c15c39b07a93392/garak/resources/ansi.py)
- [LLM Output Can Take Over Your Computer](https://interhumanagreement.substack.com/p/llm-output-can-take-over-your-computer)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in pyrit/score/true_false/regex/ by reading XSSOutputScorer and compare the pinned garak ansiescape detectors and resources referenced in the issue. The scorer work is complete when raw and escaped ANSI findings are distinguishable and covered by tests; the broader scope also includes the AnsiEscape scenario, exports, scanner documentation, dataset, and provenance entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100