aws-samples / aws-samples/sample-autonomous-cloud-coding-agents
Agent: Semantic verification advisory — scope and organicity signals
- Lingua principale
- TypeScript
- Stelle
- 143
- Fork
- 46
- Merge medio
- 3g 9h
- PR unite (30g)
- 20
Descrizione
> **Roadmap:** PR scope creep check (pr_review); Tiered validation pipeline Tier 2
> **Priority:** P1
## Component
Agent (Python runtime)
## Describe the feature
Optional post-agent **advisory** step that compares declared task intent (issue/description/PR narrative) to the actual diff and emits structured **scope and organicity** signals. Addresses the "organicity gap" from harness literature: patches can pass tests yet fail maintainer review due to style, architecture, or scope drift.
Non-blocking by default — surfaces guidance in the task record and PR body without failing the task.
## Use case
- **Reviewers** see upfront whether the agent wandered beyond the issue scope.
- **Evaluation pipeline** tracks scope creep as a failure category distinct from test failure.
- **High-risk repos** can later promote the signal to a policy gate via Cedar/workflow config.
## Proposed solution
1. New optional workflow step or post-hook phase: `assess_scope` (behind Blueprint/workflow flag, default off).
2. Structured output on task record:
```json
{
"scope_rating": "within_scope | mild_expansion | significant_expansion | likely_scope_creep",
"confidence": 0.0,
"rationale": "...",
"signals": {
"files_touched": 12,
"unrelated_deps_changed": false,
"api_surface_changed": false,
"config_changed": true
}
}
```
3. Include summary in PR body and `verification_report` (if evidence bundles shipped).
4. Implementation: deterministic diff stats + optional LLM narrative (single turn, budget-capped).
5. Share diff-analysis utilities with future `pr_review` scope check where possible.
### Acceptance criteria
- [ ] Opt-in via workflow or Blueprint flag; default profile unchanged
- [ ] Structured output persisted and API-visible
- [ ] PR body includes human-readable scope summary when enabled
- [ ] Agent tests with fixture diffs (within scope vs creep)
- [ ] Documented in `EVALUATION.md` under failure categories
## Other information
- **Related roadmap:** PR scope creep check (`pr_review`), Tier 2 code quality
- **Paper:** arXiv:2605.18747 — organicity, PatchDiff / maintainer rejection
- **Out of scope for v1:** Blocking PR creation on scope rating
## Acknowledgements
- [ ] I may be able to implement this feature
- [ ] This might be a breaking change
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia tracciando il percorso del flag opzionale di workflow o Blueprint per il passaggio proposto assess_scope, quindi esamina come vengono popolati i record delle attività, verification_report e i corpi delle PR. Esamina la relazione pr_review e EVALUATION.md prima di definire i diff delle fixture per i casi entro l'ambito e i casi di scope creep. Il lavoro è completo quando include un comportamento opt-in, un output strutturato persistito e visibile tramite API, un riepilogo della PR, test e documentazione, senza modificare il profilo predefinito.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- ai, devtools
- Tipo di issue
- Funzionalità
- Difficoltà
- 5/5
- Tempo stimato
- Più di una settimana
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 42/100