PAIR-code / PAIR-code/deliberate-lab
agents(skills): codify maintainer PR evaluation protocol and modernize eval-pr skill
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 96
- Forks
- 40
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 20
Description
Problem
The current Pull Request evaluation workflow in Deliberate Lab lacks end-to-end codification. While .agents/skills/eval-pr/ provides a mechanical 3-step checkout guide (git fetch + git worktree add), it misses the actual asynchronous maintainer evaluation lifecycle.
In practice, maintainers face several recurring frictions:
- Implicit Maintainer Workflow: The primary maintainer's end-to-end evaluation procedure (triaging PR scope, gatekeeping manual test steps, belt-and-suspenders local test execution, and behavioral walkthroughs) exists solely in the maintainer's head.
- Missing Manual Verification Gate: Contributor PRs (which are predominantly new software features) virtually always omit explicit manual testing instructions, creating friction and delays when verifying functionality locally.
- The AI Agent Communication Gap: Maintainer code reviews are increasingly carried out by or paired with the contributor's AI agent. Review feedback written as vague or conversational comments leads to multi-round thrashing; feedback must be prompt-ready, test-oriented, and unambiguous.
- Outdated Skill & Missing Invariants: The existing
eval-prskill is purely a mechanical checkout helper. It contains obsolete rules (e.g. prohibitinggit branch -D, which breaks for squash-merged remote branches) and lacks automated scaffolding for dependency hydration, local testing, and post-merge retirement (connecting to #1238's automated PR garbage collection).
Proposed Solution
Codify the maintainer evaluation protocol as a formal architecture decision record (.agents/decisions/xxxx-pr-evaluation-lifecycle.md, where xxxx is assigned to the next available number at implementation time) and modernize (or replace/split) the eval-pr skill into a comprehensive maintainer pairing workflow.
1. Architecture Decision Record (.agents/decisions/xxxx-pr-evaluation-lifecycle.md)
Formalize the 6-phase maintainer evaluation protocol into repository canon:
- Phase 1: Triage & Classification:
- Classify PR scope (Feature vs. Meta/Tooling vs. Bugfix vs. Docs).
- Trigger / approve pending GitHub Actions CI runs if maintainer approval is required.
- Phase 2: Gatekeeping Verification Pre-requisites:
- Inspect the PR body for explicit, step-by-step manual testing instructions.
- If missing (the default case), immediately request them so the contributor (and their AI agent) can draft them in parallel with local test execution.
- Phase 3: Worktree Hydration & Belt-and-Suspenders Local Verification:
- Spin up isolated sibling worktree:
pr-<number>. - Hydrate dependencies via
npm ci. - Compile packages and run lints, unit tests, and integration tests locally to verify test harness integrity independently of GitHub Actions runner quirks.
- Spin up isolated sibling worktree:
- Phase 4: Code & Architecture Review (Reviewing for the Contributor's Agent):
- Inspect diff on GitHub for asynchronous state, multi-party invariants, and test coverage.
- Formulate review comments specifically as prompt-ready, actionable instructions for the author's AI agent.
- Phase 5: Live Behavioral Walkthrough:
- Launch local server via
./run_locally.shinside thepr-<number>worktree. - Execute the author's manual testing steps; inspect for regressions and behavioral anomalies.
- Launch local server via
- Phase 6: Merge & Automated Sandbox Retirement:
- Squash & merge upstream via GitHub.
- Rely on
workspace-sync(#1238) for automated garbage collection of the evaluation worktree and local tracking branch.
2. Modernize / Refactor the eval-pr Skill (.agents/skills/eval-pr/)
Transform eval-pr from a passive 3-line git checkout guide into an active maintainer assistant:
- Scope & Gate Inspection: Inspect PR metadata (
gh pr view) to verify whether manual testing instructions are present; offer copy-pasteable request prompts if absent. - Automated Worktree Hydration: Automate
git worktree add,npm ci, build (npm run build -w utils), and test execution. - Behavioral Runner Helper: Provide clear invocation commands for
./run_locally.shwithin the specificpr-<number>context. - Align Lifecycle Rules: Update skill documentation to reflect automated terminal-state cleanup via
workspace-syncand deprecate the obsoletegit branch -drule.
Related Issues & Dependencies
- #1178 (Automating manual testing step generation via GitHub Actions / AI)
- #1238 (Automated garbage collection of merged/closed
pr-<number>worktrees inworkspace-sync) - #1243 (Adoption of
.agents/decisionsframework) - #1245 (
ghskill helpers for non-truncating PR inspection)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read the existing .agents/skills/eval-pr/ skill and .agents/decisions framework first, then review related issues #1238, #1243, #1245, and #1178. Trace the current gh pr view, git worktree, npm ci, npm run build -w utils, test, and ./run_locally.sh entry points. Done means the six-phase lifecycle is recorded and the skill covers inspection, hydration, verification, walkthrough, and cleanup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, github, github-actions, shell
- Domain
- developer-experience, documentation, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100