aws-samples / aws-samples/sample-autonomous-cloud-coding-agents
feat(orchestration): PR outcome tracking via GitHub webhooks
- Dominant language
- TypeScript
- Stars
- 143
- Forks
- 46
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 20
Description
**Context:** ROADMAP.md → Agent quality → PR outcome tracking
**Related:** #27 (review feedback memory), #412 (stats), evaluation pipeline draft
---
## Component
API or orchestration
## Describe the feature
Track PR **merge / close / reject** outcomes via GitHub webhooks (and analogous providers later). Persist positive/negative signals on the task record and feed evaluation pipeline and memory systems.
## Use case
Task `COMPLETED` today means "PR opened," not "change landed." Operators cannot measure true success rate, revision cycles, or memory effectiveness without outcome linkage.
## Proposed solution
1. Extend GitHub webhook processor for `pull_request.closed` (merged vs not).
2. Link outcome to originating `task_id` (branch name, PR metadata, or stored `pr_url`).
3. Update task record: `pr_outcome: merged | closed | abandoned`, `merged_at`, `revision_count` (if trackable).
4. Emit `pr_outcome_recorded` to `TaskEventsTable`.
5. Expose in `bgagent stats` / evaluation APIs (#412).
## Other information
- Feeds **Review feedback memory loop** (#27) and **A/B prompt experiments** draft.
- Design context: `docs/design/EVALUATION.md`.
- [ ] This might be a breaking change
Contributor guide
Research direction
Start with docs/design/EVALUATION.md and the GitHub webhook processor. Trace how task records and TaskEventsTable are represented, then review the stats and evaluation API references in #412. Done means closed pull request outcomes are linked to tasks, persisted with the specified signals, emitted as events, and exposed to stats or evaluation consumers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github, typescript
- Domain
- api, backend, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100