NVIDIA-NeMo / NVIDIA-NeMo/Guardrails
feature: an action-layer rail for agent tool calls (secret leak, data exfiltration, malicious tool use)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 843
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 25
Description
Did you check the docs?
- I have read all the NeMo-Guardrails docs
Is your feature request related to a problem? Please describe.
Yes. NeMo Guardrails' built-in library has deep coverage of the conversational text layer — input/output moderation, topic control, RAG fact-checking and hallucination, PII (Presidio / Private AI), and jailbreak heuristics. Every one of those rails inspects message text: user_input or bot_response.
As NeMo gets pointed at agents rather than chatbots, the risky surface moves from what the model says to what the agent does — the tool calls it issues and the tool outputs it ingests. There's no rail that inspects a tool call before it executes. These failure modes are about action shape, not message content, so text-layer rails structurally can't catch them:
A fetched web page carries injected instructions; the agent pipes it into a shell / code-execution tool. Code runs before any content-safety rail sees a problem.
A tool call carries a live credential or API key in its arguments. PII rails flag names and SSNs in prose, not secrets in a tool-call payload.
Tool output is routed to an outbound channel (messaging, webhook, file write) that exfiltrates data the agent has handled.
In web3 contexts, a tool call signs or sends a transaction that no text rail evaluates.
There's currently no extension point to allow, block, or require confirmation on a tool call based on its content, the way input/output rails already do for messages.
Describe the solution you'd like
A community rail that inspects tool calls and tool outputs, not just message text, and returns a structured verdict: allow | block | require_user_confirm with a reason.
Concretely, mirroring the existing community-rail pattern (a custom action + a Colang flow + a docs/user-guides/community/ page):
colang
define flow agentguard check tool call
$verdict = execute agentguard_check_action
if $verdict.action == "block"
bot refuse tool call
stop
if $verdict.action == "require_user_confirm"
bot ask user confirmation
stop
The action calls a local, deterministic decision engine over a server_endpoint (the same config shape AlignScore and the jailbreak-detection server already use). It runs locally with no API key and no hosted dependency, so it fits "Community Models and Libraries" next to Presidio rather than the third-party-API rails. It runs at roughly 0.13 ms per call on an open benchmark, which is what makes per-tool-call inspection viable in the critical path where an 8B safety-model round-trip would not be.
I'm offering to implement this myself. If NeMo already surfaces pre-execution tool-call events to rails, it's a pure community add (action + flow + docs, no core changes). If it doesn't yet, the rail needs a small core hook — fire an event with tool_name + args before dispatch and let a rail act on the return — and I'm happy to implement both the hook and the rail. This same engine already runs in production against two other agent platforms (Claude Code PreToolUse/PostToolUse, OpenClaw before_tool_call/after_tool_call) on a shared adapter, so a NeMo rail would be a third instance of an existing pattern.
Describe alternatives you've considered
-
Self-check input/output rails on tool-call text. Prompting the main LLM to judge a serialized tool call adds an LLM round-trip per call (latency and cost), and detection quality depends on the same model that issued the call. Unreliable for a security boundary, and far too slow on the action path.
-
Jailbreak / perplexity heuristics. These score chat text for prefix/suffix attacks. They say nothing about whether a tool call leaks a credential, exfiltrates data, or executes fetched content.
-
PII rails (Presidio / Private AI). Excellent for names, emails, SSNs in prose. They don't model secrets in tool arguments, malicious command shape, or exfiltration routing, and they don't gate execution.
-
An external proxy in front of tool/MCP calls. Works for MCP-routed tools but misses native tool calls and memory writes, adds a process boundary and latency, and lives outside the guardrails config so it can't be reasoned about alongside the other rails.
-
Patch each host integration separately. Doesn't scale across LangChain, the server, and Colang actions. A first-class rail lets one config cover the action layer the way input/output rails already cover the text layer.
Additional context
- AgentGuard is MIT and open source: https://github.com/GoPlusSecurity/agentguard (@goplus/agentguard). The decision engine covers the agent action surface — malicious tool calls, data exfiltration, secret/credential leak, web3 threats, and human-in-the-loop bypass — categories that sit outside the conversational-text coverage the library has today. It's intended as additive: content moderation, topic control, and RAG checks stay exactly as they are.
- Benchmark (open, reproducible): [link]. ~0.13 ms average per call, which is the property that makes it usable per-tool-call rather than per-conversation.
- One implementation note for reviewers: NeMo Guardrails is Python; the engine ships as Node/npm plus an MCP server. The integration calls it over a local HTTP sidecar (the AlignScore / jailbreak-server pattern), so there's no cross-language runtime dependency pulled into NeMo.
- I'd bring a runnable LangChain demo (a fetched-page-to-shell pivot blocked by the rail) to whichever integration path you prefer
For contact: atakan@gopluslabs.io
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing community-rail pattern and the AlignScore/jailbreak server configuration to determine whether pre-execution tool-call events are already exposed. Define the integration around tool_name and args, then verify that allow, block, and require_user_confirm verdicts gate tool execution and that the local server path works with a runnable LangChain demo.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, python
- Domain
- ai, backend, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100