aporthq / aporthq/aport-agent-guardrails
Policy Pack Proposal: deliverable.task.evidence_groundtruth.v1
- Dominant language
- Shell
- Stars
- 25
- Forks
- 4
- Avg merge
- 4h 14m
- Merged PRs (30d)
- 10
Description
## Policy Pack Proposal
**Policy ID:** `deliverable.task.evidence_groundtruth.v1`
**Description:**
Requires that completion/status claims made by an agent (e.g. "tests pass", "rollback is impossible", "criteria met") reference independently checkable ground truth — not just a free-text attestation. The existing `deliverable.task.complete.v1` pack requires a non-empty `evidence` string per criterion, but does not verify that the claimed evidence is actually true. This pack adds that verification step.
**Tools Covered:**
- `deliverable.task.complete.v1` (companion/extension)
- `code.repository.merge.v1`
- any tool call where an agent self-reports task or rollback status
**Limits:**
- Ground truth reference required for: rollback status claims, "tests passing" claims, destructive-operation completions
- Accepted reference types: `ci_run_id`, `file_hash`, `command_exit_code`, `external_url_status`
- Unverifiable reference → deny (`oap.evidence_unverifiable`)
**Use Case:**
In July 2025, an agent (Replit) deleted a production database despite an active code freeze, then falsely reported that a rollback was impossible. `deliverable.task.complete.v1` would have accepted this claim, since it only checks that an evidence string is non-empty — not that it's true. This pack closes that gap by requiring a checkable reference instead of a free-text claim.
**Example Context:**
```jason
{
"task_id": "task_123",
"output_type": "code",
"criteria_attestations": [
{
"criterion_id": "rollback_status",
"met": true,
"evidence": "rollback impossible",
"ground_truth_ref": {
"type": "ci_run_id",
"value": "run_9f8a3b"
}
}
]
}
```
**References:**
- Relates to existing `deliverable.task.complete.v1` (same repo)
I have a working reference implementation (JSON pack + evaluator + conformance test cases, 5/5 passing against a Replit-incident-style test case) ready to include in a PR once there's agreement on direction. Open to this being a standalone pack or an extension flag on `deliverable.task.complete.v1` — happy to go either way.
Contributor guide
Assessment
This issue has not been assessed yet.