PAIR-code / PAIR-code/deliberate-lab

agents(skills): codify maintainer PR evaluation protocol and modernize eval-pr skill

Open
#1,247 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area:workspace
Dominant language
TypeScript
Stars
96
Forks
40
Avg merge
2d 11h
Merged PRs (30d)
20

Description

Problem

The current Pull Request evaluation workflow in Deliberate Lab lacks end-to-end codification. While .agents/skills/eval-pr/ provides a mechanical 3-step checkout guide (git fetch + git worktree add), it misses the actual asynchronous maintainer evaluation lifecycle.

In practice, maintainers face several recurring frictions:

  1. Implicit Maintainer Workflow: The primary maintainer's end-to-end evaluation procedure (triaging PR scope, gatekeeping manual test steps, belt-and-suspenders local test execution, and behavioral walkthroughs) exists solely in the maintainer's head.
  2. Missing Manual Verification Gate: Contributor PRs (which are predominantly new software features) virtually always omit explicit manual testing instructions, creating friction and delays when verifying functionality locally.
  3. The AI Agent Communication Gap: Maintainer code reviews are increasingly carried out by or paired with the contributor's AI agent. Review feedback written as vague or conversational comments leads to multi-round thrashing; feedback must be prompt-ready, test-oriented, and unambiguous.
  4. Outdated Skill & Missing Invariants: The existing eval-pr skill is purely a mechanical checkout helper. It contains obsolete rules (e.g. prohibiting git branch -D, which breaks for squash-merged remote branches) and lacks automated scaffolding for dependency hydration, local testing, and post-merge retirement (connecting to #1238's automated PR garbage collection).

Proposed Solution

Codify the maintainer evaluation protocol as a formal architecture decision record (.agents/decisions/xxxx-pr-evaluation-lifecycle.md, where xxxx is assigned to the next available number at implementation time) and modernize (or replace/split) the eval-pr skill into a comprehensive maintainer pairing workflow.

1. Architecture Decision Record (.agents/decisions/xxxx-pr-evaluation-lifecycle.md)

Formalize the 6-phase maintainer evaluation protocol into repository canon:

  • Phase 1: Triage & Classification:
    • Classify PR scope (Feature vs. Meta/Tooling vs. Bugfix vs. Docs).
    • Trigger / approve pending GitHub Actions CI runs if maintainer approval is required.
  • Phase 2: Gatekeeping Verification Pre-requisites:
    • Inspect the PR body for explicit, step-by-step manual testing instructions.
    • If missing (the default case), immediately request them so the contributor (and their AI agent) can draft them in parallel with local test execution.
  • Phase 3: Worktree Hydration & Belt-and-Suspenders Local Verification:
    • Spin up isolated sibling worktree: pr-<number>.
    • Hydrate dependencies via npm ci.
    • Compile packages and run lints, unit tests, and integration tests locally to verify test harness integrity independently of GitHub Actions runner quirks.
  • Phase 4: Code & Architecture Review (Reviewing for the Contributor's Agent):
    • Inspect diff on GitHub for asynchronous state, multi-party invariants, and test coverage.
    • Formulate review comments specifically as prompt-ready, actionable instructions for the author's AI agent.
  • Phase 5: Live Behavioral Walkthrough:
    • Launch local server via ./run_locally.sh inside the pr-<number> worktree.
    • Execute the author's manual testing steps; inspect for regressions and behavioral anomalies.
  • Phase 6: Merge & Automated Sandbox Retirement:
    • Squash & merge upstream via GitHub.
    • Rely on workspace-sync (#1238) for automated garbage collection of the evaluation worktree and local tracking branch.
2. Modernize / Refactor the eval-pr Skill (.agents/skills/eval-pr/)

Transform eval-pr from a passive 3-line git checkout guide into an active maintainer assistant:

  • Scope & Gate Inspection: Inspect PR metadata (gh pr view) to verify whether manual testing instructions are present; offer copy-pasteable request prompts if absent.
  • Automated Worktree Hydration: Automate git worktree add, npm ci, build (npm run build -w utils), and test execution.
  • Behavioral Runner Helper: Provide clear invocation commands for ./run_locally.sh within the specific pr-<number> context.
  • Align Lifecycle Rules: Update skill documentation to reflect automated terminal-state cleanup via workspace-sync and deprecate the obsolete git branch -d rule.

Related Issues & Dependencies
  • #1178 (Automating manual testing step generation via GitHub Actions / AI)
  • #1238 (Automated garbage collection of merged/closed pr-<number> worktrees in workspace-sync)
  • #1243 (Adoption of .agents/decisions framework)
  • #1245 (gh skill helpers for non-truncating PR inspection)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the existing .agents/skills/eval-pr/ skill and .agents/decisions framework first, then review related issues #1238, #1243, #1245, and #1178. Trace the current gh pr view, git worktree, npm ci, npm run build -w utils, test, and ./run_locally.sh entry points. Done means the six-phase lifecycle is recorded and the skill covers inspection, hydration, verification, walkthrough, and cleanup.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, github, github-actions, shell
Domain
developer-experience, documentation, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.