aporthq / aporthq/aport-agent-guardrails

Policy Pack Proposal: deliverable.task.evidence_groundtruth.v1

Open
#90 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Shell
Stars
25
Forks
4
Avg merge
4h 14m
Merged PRs (30d)
10

Description

## Policy Pack Proposal

**Policy ID:** `deliverable.task.evidence_groundtruth.v1`

**Description:**
Requires that completion/status claims made by an agent (e.g. "tests pass", "rollback is impossible", "criteria met") reference independently checkable ground truth — not just a free-text attestation. The existing `deliverable.task.complete.v1` pack requires a non-empty `evidence` string per criterion, but does not verify that the claimed evidence is actually true. This pack adds that verification step.

**Tools Covered:**
- `deliverable.task.complete.v1` (companion/extension)
- `code.repository.merge.v1`
- any tool call where an agent self-reports task or rollback status

**Limits:**
- Ground truth reference required for: rollback status claims, "tests passing" claims, destructive-operation completions
- Accepted reference types: `ci_run_id`, `file_hash`, `command_exit_code`, `external_url_status`
- Unverifiable reference → deny (`oap.evidence_unverifiable`)

**Use Case:**
In July 2025, an agent (Replit) deleted a production database despite an active code freeze, then falsely reported that a rollback was impossible. `deliverable.task.complete.v1` would have accepted this claim, since it only checks that an evidence string is non-empty — not that it's true. This pack closes that gap by requiring a checkable reference instead of a free-text claim.

**Example Context:**
```jason
{
"task_id": "task_123",
"output_type": "code",
"criteria_attestations": [
{
"criterion_id": "rollback_status",
"met": true,
"evidence": "rollback impossible",
"ground_truth_ref": {
"type": "ci_run_id",
"value": "run_9f8a3b"
}
}
]
}

```

**References:**
- Relates to existing `deliverable.task.complete.v1` (same repo)

I have a working reference implementation (JSON pack + evaluator + conformance test cases, 5/5 passing against a Replit-incident-style test case) ready to include in a PR once there's agreement on direction. Open to this being a standalone pack or an extension flag on `deliverable.task.complete.v1` — happy to go either way.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.