Structured output mode — constrain agent responses to a declared JSON schema
- Dominant language
- Go
- Stars
- 5.1k
- Forks
- 541
- Avg merge
- 5h 46m
- Merged PRs (30d)
- 760
Description
### Problem
Agentic workflows currently produce freeform text in their agent output. Downstream jobs that consume this output must parse unstructured natural language, which is fragile, non-deterministic, and breaks when the LLM changes phrasing. There is no built-in mechanism to declare that an agent's response must conform to a specific JSON schema, even though the underlying models (OpenAI, Anthropic, Gemini) all support constrained/structured output natively.
This creates a gap: workflows that need machine-readable output from the agent job — for routing, aggregation, database writes, or chaining into subsequent agent jobs — must resort to brittle regex parsing or prompt-engineering hacks (`"respond only in JSON"`), with no schema validation and no compile-time safety.
### Why this matters for agentic workflows
The `safe-outputs` feature already demonstrates that gh-aw understands the need for structured side-effects (comments, PRs, labels). Structured output mode extends this principle to the agent's **primary response** — the content the agent produces, not just the actions it takes.
### Use Cases
1. **Multi-agent pipelines**: Agent A produces a structured analysis → Agent B consumes it as typed input. Today this requires an intermediate parsing job and fails silently on schema drift.
2. **Data extraction workflows**: An agent reads an issue, extracts fields (severity, component, repro steps) into a JSON object, and a downstream job writes them to a project board or database. Without schema enforcement, missing fields cause silent data loss.
3. **Decision routing**: An agent evaluates a PR and outputs `{ "decision": "APPROVE" | "REQUEST_CHANGES" | "ESCALATE", "reasoning": "..." }`. The downstream job routes based on `decision`. Freeform text makes this routing unreliable.
4. **Aggregation across matrix jobs**: Combined with [#26598](https://github.com/github/gh-aw/issues/26598) (matrix strategy), parallel agent instances each produce a typed result object. A fan-in job aggregates them. Without schema enforcement, one malformed response breaks the entire aggregation.
5. **Audit and compliance**: Regulated industries need deterministic, schema-validated output for audit trails — not "the LLM usually returns JSON."
### Proposed Solution
Add a `structured-output` configuration in the workflow frontmatter that declares a JSON schema the agent's final response must conform to.
#### Frontmatter syntax
```yaml
---
on:
issues:
types: [labeled]
engine:
id: copilot
structured-output:
schema:
type: object
properties:
decision:
type: string
enum: [APPROVE, REQUEST_CHANGES, ESCALATE]
reasoning:
type: string
minLength: 10
confidence:
type: number
minimum: 0
maximum: 1
required: [decision, reasoning, confidence]
additionalProperties: false
tools:
github:
toolsets: [default]
---
```
#### Alternative: schema file reference
For larger schemas, reference an external file:
```yaml
structured-output:
schema-file: .github/schemas/triage-output.schema.json
```
#### Runtime behavior
1. **Compile-time**: `gh aw compile` validates the schema is well-formed JSON Schema (draft-07 or 2020-12).
2. **Agent invocation**: The runtime passes the schema to the underlying LLM using its native structured output mechanism:
- OpenAI: `response_format: { type: "json_schema", json_schema: { ... } }`
- Anthropic: Tool use with a single tool matching the schema
- Other engines: Prompt-based enforcement with post-hoc validation as fallback
3. **Validation**: After the agent responds, the runtime validates the response against the declared schema before making it available to downstream jobs.
4. **Failure mode**: If validation fails, the runtime retries once with an error-correction prompt. If validation fails again, the workflow fails with a clear schema-violation error (not a silent pass-through of malformed data).
#### Output access in downstream jobs
The validated JSON is available as a typed output:
```yaml
jobs:
route:
runs-on: ubuntu-latest
needs: agent
steps:
- run: |
DECISION='${{ needs.agent.outputs.structured.decision }}'
if [ "$DECISION" = "ESCALATE" ]; then
gh issue edit $ISSUE --add-label "needs-human"
fi
```
### Prior Art
- **`safe-outputs`** already constrains agent side-effects (add-comment, create-issue, etc.) with typed schemas. Structured output mode is the read-side complement — constraining the agent's response content, not just its actions.
- **[#28863](https://github.com/github/gh-aw/issues/28863)** (Large content not passed to output job) highlights existing friction in the output pipeline. Structured output would benefit from a robust output transport regardless of size.
- **[#26598](https://github.com/github/gh-aw/issues/26598)** (Matrix strategy) — structured output pairs naturally with matrix fan-out/fan-in patterns.
- **OpenAI Structured Outputs** (GA since Aug 2024): `response_format: { type: "json_schema" }` with 100% schema adherence guarantee.
- **Anthropic tool_use**: Constrained output via single-tool invocation with JSON schema.
- **LangChain/LangGraph**: `with_structured_output()` is the standard pattern for schema-constrained LLM responses.
### Alternatives Considered
| Approach | Limitation |
|----------|-----------|
| Prompt-engineer "respond in JSON" | No enforcement; LLM can drift, wrap in markdown, or omit fields |
| Parse output with `jq` in downstream job | Fails silently on schema drift; no retry; no compile-time safety |
| Custom job that validates + retries | Adds a full extra job layer per workflow; not first-class |
| Use `safe-outputs` as a proxy | Safe outputs are for side-effects, not the agent's primary response; shoehorning data into a comment or file is a workaround |
| External validation MCP server | Requires extra infra; moves schema definition away from the workflow |
### Scope Questions for Maintainers
1. **Schema format**: JSON Schema draft-07 (widely supported) or 2020-12 (latest)?
2. **Engine coverage**: Start with Copilot/OpenAI only (native structured output), or include Claude/Codex with fallback validation from day one?
3. **Retry policy**: Single retry with error-correction prompt, or configurable (`max-retries: 3`)?
4. **Schema composition**: Should `structured-output` support `$ref` to shared schema definitions, or keep it self-contained per workflow?
5. **Interaction with `safe-outputs`**: Can a workflow declare both structured output (for the response body) and safe outputs (for side-effects) simultaneously?
### Additional Context
This feature would make gh-aw competitive with framework-level structured output support (LangChain, CrewAI, AutoGen) while staying native to the GitHub Actions paradigm. The `safe-outputs` design already demonstrates that gh-aw can enforce output contracts — structured output extends that contract to the agent's primary response.
Contributor guide
Assessment
This issue has not been assessed yet.