microsoft / microsoft/conductor
Feature: Output Guardrails & Validation Hooks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 448
- Forks
- 65
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
Summary
Add a guardrails section to agent definitions for semantic output validation beyond JSON schema type checking — including regex patterns, length limits, and custom script-based checks.
Motivation
Research shows frontier models spontaneously exhibit deceptive behaviors in multi-agent settings (UC Berkeley/UC Santa Cruz study), 30-50% of AI agents bypass ethical constraints under KPI pressure, and RAG document poisoning can cause fabricated financial data. Conductor validates output types today (JSON schema) but has no way to validate output content or semantics.
Proposed Design
agents:
- name: financial_analyst
model: gpt-5.2
output:
recommendation:
type: string
guardrails:
- type: regex_deny
pattern: "(?i)(guaranteed|risk.free|100%)"
message: "Output contains prohibited financial claims"
- type: regex_require
pattern: "(?i)(disclaimer|risk)"
message: "Output must include risk disclaimer"
- type: max_length
chars: 5000
- type: custom_script
command: "python validate_output.py"
# stdin: agent output JSON
# exit 0 = pass, exit 1 = fail (stderr = failure message)
Behavior on Failure
- Guardrail failure triggers agent re-run with violation feedback injected into prompt
- Configurable
max_guardrail_retries(default: 2) before hard failure - Events emitted:
guardrail_check,guardrail_pass,guardrail_fail - Works with retry policies (#80) — guardrail retry is separate from provider error retry
Built-in Guardrail Types
| Type | Description |
|---|---|
regex_deny |
Fail if output matches pattern |
regex_require |
Fail if output does NOT match pattern |
max_length |
Fail if output exceeds character limit |
min_length |
Fail if output is below character limit |
json_schema |
Validate against an additional JSON schema (beyond output type) |
custom_script |
Run external script, pass output via stdin, check exit code |
Why It Fits Conductor
- Declarative, YAML-expressible — no code changes needed per workflow
- Script-based guardrails reuse existing
scriptstep infrastructure - Pairs with retry policies (#80) — guardrail violation → retry with feedback context
- Essential for regulated industries (finance, healthcare) adopting conductor
Effort Estimate
Medium — new validation layer in AgentExecutor post-output, new schema fields, script runner reuse from existing script step infrastructure.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the AgentExecutor post-output path, the agent-definition schema, and the existing script step infrastructure mentioned in the issue. Trace how retry policies and events are represented; done means the listed guardrail types, retry behavior, and guardrail events are integrated without disrupting existing provider-error retries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, yaml
- Domain
- backend, cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100