GoogleCloudPlatform / GoogleCloudPlatform/generative-ai

responsible-ai: add a deterministic (regex) detection pass over agent/tool inputs as a complement to the existing mitigation notebooks

Open
#2,949 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
17.7k
Forks
4.5k
Avg merge
12h 38m
Merged PRs (30d)
42

Description

The two LLM-security notebooks in `gemini/responsible-ai/` (`gemini_prompt_attacks_mitigation_examples.ipynb` and `react_rag_attacks_mitigations_examples.ipynb`) are great catalogs of prompt-attack mitigations, and both already lean on some inline deterministic checks (regex URL/schema validation, DLP pattern matching) alongside the model-side and LLM-validator layers.

One layer that isn't shown as a standalone, reusable example is a **deterministic, regex-based first-pass detection over agent/tool inputs** using an external, openly-maintained ruleset — the cheap, no-model-call filter that runs before the heavier validators. It complements the existing notebooks rather than overlapping: same defense-in-depth story, but the "rules as data" layer made explicit and portable.

I'd like to contribute a small, self-contained notebook (built from `notebook_template.ipynb`) that runs a deterministic detection pass over example agent/tool inputs and shows where it sits relative to the existing mitigation stack — using ATR (Agent Threat Rules), an open MIT ruleset for agent prompt-injection/attack patterns. It would be vendor-neutral, reference (not modify) the existing two notebooks, and pass the repo's `scripts/format.sh` + check-spelling gates. I'll also sign the Google CLA.

Before I open a PR: would you prefer a **new standalone notebook** in `gemini/responsible-ai/`, or a single added cell inside the existing ReAct/RAG notebook? Happy to go whichever way fits the folder's conventions. Tagging @tzvetkov75 as the author of the existing notebooks for a sanity check on placement.

Disclosure: I maintain ATR.

Contributor guide

Open the contributing guide

Research direction

Start with gemini/responsible-ai/notebook_template.ipynb and inspect the existing gemini_prompt_attacks_mitigation_examples.ipynb and react_rag_attacks_mitigations_examples.ipynb for folder conventions and mitigation-stack context. Add the ATR-based deterministic pass over example agent/tool inputs, reference the existing notebooks, and run scripts/format.sh plus the repository's check-spelling gate. Done means the placement choice is resolved and the notebook presents the reusable rules-as-data layer without modifying the existing notebooks.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
ai, security
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.