GoogleCloudPlatform / GoogleCloudPlatform/generative-ai
responsible-ai: add a deterministic (regex) detection pass over agent/tool inputs as a complement to the existing mitigation notebooks
- Dominant language
- Jupyter Notebook
- Stars
- 17.7k
- Forks
- 4.5k
- Avg merge
- 12h 38m
- Merged PRs (30d)
- 42
Description
The two LLM-security notebooks in `gemini/responsible-ai/` (`gemini_prompt_attacks_mitigation_examples.ipynb` and `react_rag_attacks_mitigations_examples.ipynb`) are great catalogs of prompt-attack mitigations, and both already lean on some inline deterministic checks (regex URL/schema validation, DLP pattern matching) alongside the model-side and LLM-validator layers.
One layer that isn't shown as a standalone, reusable example is a **deterministic, regex-based first-pass detection over agent/tool inputs** using an external, openly-maintained ruleset — the cheap, no-model-call filter that runs before the heavier validators. It complements the existing notebooks rather than overlapping: same defense-in-depth story, but the "rules as data" layer made explicit and portable.
I'd like to contribute a small, self-contained notebook (built from `notebook_template.ipynb`) that runs a deterministic detection pass over example agent/tool inputs and shows where it sits relative to the existing mitigation stack — using ATR (Agent Threat Rules), an open MIT ruleset for agent prompt-injection/attack patterns. It would be vendor-neutral, reference (not modify) the existing two notebooks, and pass the repo's `scripts/format.sh` + check-spelling gates. I'll also sign the Google CLA.
Before I open a PR: would you prefer a **new standalone notebook** in `gemini/responsible-ai/`, or a single added cell inside the existing ReAct/RAG notebook? Happy to go whichever way fits the folder's conventions. Tagging @tzvetkov75 as the author of the existing notebooks for a sanity check on placement.
Disclosure: I maintain ATR.
Contributor guide
Research direction
Start with gemini/responsible-ai/notebook_template.ipynb and inspect the existing gemini_prompt_attacks_mitigation_examples.ipynb and react_rag_attacks_mitigations_examples.ipynb for folder conventions and mitigation-stack context. Add the ATR-based deterministic pass over example agent/tool inputs, reference the existing notebooks, and run scripts/format.sh plus the repository's check-spelling gate. Done means the placement choice is resolved and the notebook presents the reusable rules-as-data layer without modifying the existing notebooks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook
- Domain
- ai, security
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100