aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

feat(orchestration): PR outcome tracking via GitHub webhooks

Open
#473 0 comments 0 reactions 0 assignees View on GitHub
adapters enhancement orchestration
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

**Context:** ROADMAP.md → Agent quality → PR outcome tracking
**Related:** #27 (review feedback memory), #412 (stats), evaluation pipeline draft

---

## Component

API or orchestration

## Describe the feature

Track PR **merge / close / reject** outcomes via GitHub webhooks (and analogous providers later). Persist positive/negative signals on the task record and feed evaluation pipeline and memory systems.

## Use case

Task `COMPLETED` today means "PR opened," not "change landed." Operators cannot measure true success rate, revision cycles, or memory effectiveness without outcome linkage.

## Proposed solution

1. Extend GitHub webhook processor for `pull_request.closed` (merged vs not).
2. Link outcome to originating `task_id` (branch name, PR metadata, or stored `pr_url`).
3. Update task record: `pr_outcome: merged | closed | abandoned`, `merged_at`, `revision_count` (if trackable).
4. Emit `pr_outcome_recorded` to `TaskEventsTable`.
5. Expose in `bgagent stats` / evaluation APIs (#412).

## Other information

- Feeds **Review feedback memory loop** (#27) and **A/B prompt experiments** draft.
- Design context: `docs/design/EVALUATION.md`.

- [ ] This might be a breaking change

Contributor guide

Open the contributing guide

Research direction

Start with docs/design/EVALUATION.md and the GitHub webhook processor. Trace how task records and TaskEventsTable are represented, then review the stats and evaluation API references in #412. Done means closed pull request outcomes are linked to tasks, persisted with the specified signals, emitted as events, and exposed to stats or evaluation consumers.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, typescript
Domain
api, backend, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.