theam / theam/facility

Outcomes end at merge: add a post-merge window so "accepted" is joined to "held"

Open
#286 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
71
Forks
64
Avg merge
15h 38m
Merged PRs (30d)
66

Description

What the collector measures today

services/api/src/watchtower/outcomes.ts joins each agent PR to fate, acceptance (human squash-merge with evidence), review rounds, and fixup commits. oneShot = merged with zero change requests and zero human commits after the first bot commit on the PR branch.

Everything stops at merged_at. There is no signal for what happened to the change afterwards.

What that misses
  • A revert of the merge commit (the sdlc page reports "0 production reverts"; nothing here measures it).
  • A human commit or PR touching the same paths within N days of merge. Fixups landed in a separate PR are invisible to fixupCommits, so one-shot is overstated.
  • A platform issue / CI-doctor fingerprint whose first appearance follows the merge and touches the same files.

The ROADMAP's definition of done, item 8, says deployment and production signals are joined back to the PR and outcome. The outcome record has nowhere to hold them.

Proposal

Add a post-merge evaluation to the nightly outcomes job:

  • New columns on outcomes: held (bool, null until the window closes), post_merge_window_days (project setting, default 7), reverted_at, post_merge_human_commits (count of non-bot commits on the default branch touching the PR's changed paths inside the window), linked_incident_fingerprint.
  • held = merged && !reverted && post_merge_human_commits == 0 && no linked incident.
  • Analytics: report held next to accepted and oneShot in analytics.ts and the project page, with unassessed while the window is open.
  • Keep the evidence path independent: read the default branch and revert commits from the GitHub API, never from Facility's own events, per the watchtower's rule 1.
Why

Acceptance measures whether a reviewer said yes. Held measures whether they were right. The learning agent's "silence outranks everything" rule applies here: a change that quietly needed a follow-up fix is the failure class most worth clustering, and today it never turns anything red.

I can open a PR against outcomes.ts and analytics.ts with tests if the shape is agreed. Filing first per CONTRIBUTING.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the proposed changes in services/api/src/watchtower/outcomes.ts and analytics.ts, then read ROADMAP item 8 and CONTRIBUTING for the project’s constraints. Define the post-merge fields and window behavior, add tests as proposed, and verify that analytics and the project page report held or unassessed alongside accepted and oneShot.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, typescript
Domain
analytics, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.