aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

Agent: Semantic verification advisory — scope and organicity signals

Offen
#514 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
agent-runtime enhancement P1
Vorherrschende Sprache
TypeScript
Sterne
143
Forks
46
Ø Merge
3 T. 10 Std.
Gemergte PRs (30 T.)
24

Beschreibung

> **Roadmap:** PR scope creep check (pr_review); Tiered validation pipeline Tier 2
> **Priority:** P1

## Component

Agent (Python runtime)

## Describe the feature

Optional post-agent **advisory** step that compares declared task intent (issue/description/PR narrative) to the actual diff and emits structured **scope and organicity** signals. Addresses the "organicity gap" from harness literature: patches can pass tests yet fail maintainer review due to style, architecture, or scope drift.

Non-blocking by default — surfaces guidance in the task record and PR body without failing the task.

## Use case

- **Reviewers** see upfront whether the agent wandered beyond the issue scope.
- **Evaluation pipeline** tracks scope creep as a failure category distinct from test failure.
- **High-risk repos** can later promote the signal to a policy gate via Cedar/workflow config.

## Proposed solution

1. New optional workflow step or post-hook phase: `assess_scope` (behind Blueprint/workflow flag, default off).

2. Structured output on task record:

```json
{
"scope_rating": "within_scope | mild_expansion | significant_expansion | likely_scope_creep",
"confidence": 0.0,
"rationale": "...",
"signals": {
"files_touched": 12,
"unrelated_deps_changed": false,
"api_surface_changed": false,
"config_changed": true
}
}
```

3. Include summary in PR body and `verification_report` (if evidence bundles shipped).

4. Implementation: deterministic diff stats + optional LLM narrative (single turn, budget-capped).

5. Share diff-analysis utilities with future `pr_review` scope check where possible.

### Acceptance criteria

- [ ] Opt-in via workflow or Blueprint flag; default profile unchanged
- [ ] Structured output persisted and API-visible
- [ ] PR body includes human-readable scope summary when enabled
- [ ] Agent tests with fixture diffs (within scope vs creep)
- [ ] Documented in `EVALUATION.md` under failure categories

## Other information

- **Related roadmap:** PR scope creep check (`pr_review`), Tier 2 code quality
- **Paper:** arXiv:2605.18747 — organicity, PatchDiff / maintainer rejection
- **Out of scope for v1:** Blocking PR creation on scope rating

## Acknowledgements

- [ ] I may be able to implement this feature
- [ ] This might be a breaking change

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne damit, den Pfad des optionalen Workflow- oder Blueprint-Flags für den vorgeschlagenen Schritt assess_scope nachzuverfolgen, und untersuche anschließend, wie Task-Datensätze, verification_report und PR-Bodies befüllt werden. Prüfe die pr_review-Beziehung und EVALUATION.md, bevor du Fixture-Diffs für Fälle innerhalb des Scopes und für Scope-Creep-Fälle definierst. Erledigt bedeutet Opt-in-Verhalten, persistierte, über die API sichtbare strukturierte Ausgabe, eine PR-Zusammenfassung, Tests und Dokumentation, ohne das Standardprofil zu ändern.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
python
Bereich
ai, devtools
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
42/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.