addyosmani / addyosmani/agent-skills

Proposal: HUMAN.md, an evidence-based human protocol for each skill

Open
#529 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
93.8k
Forks
10k
Avg merge
3d 15h
Merged PRs (30d)
23

Description

Every SKILL.md in this pack instructs the agent. Nothing instructs the human at the gates the skill itself creates. This proposes the counterpart file: HUMAN.md, next to SKILL.md, never loaded by the agent.

Here is what one would look like (test-driven-development, abridged):

## Gate 1: the failing test (after "Step 1: RED", ~2 min)
Read only the test diff, then the failure output. Ask:
- Does it fail for the right reason? (read the error message, not the code)
- Would a trivially wrong implementation pass it?
→ redirect: "Strengthen the assertion, it currently accepts X."
- Behavior, or implementation detail?
→ redirect: "Rewrite the test against the public interface only."

## Gate 2: after the refactor, before commit (after "Step 3: REFACTOR", ~3 min)
Read the implementation diff only. Ask:
- Did anything change outside the slice?
→ redirect: "Revert changes outside the task scope."
- Do the test names still describe behavior after the refactor?

## Stop rules
- Third failed iteration on the same test → write the test yourself.
- Diff grew beyond the files named in the plan → pause and re-scope.

That is the whole format: gates that cite SKILL.md step headings, gate count scaled by how often that skill's class of work fails, a review phrased as questions (~5 min total per run), redirects the agent can act on, objective stop rules.

**Why a separate file, not a `## For humans` section:** SKILL.md is loaded into agent context in full, so a human section would cost tokens on every invocation and put the reviewer's checklist inside the model's prompt. A separate file costs zero tokens, and since no harness parses it, it is tool-agnostic by construction.

**Why not the existing `## Verification` sections:** every skill already ends with one, but it is a list of exit criteria the agent self-certifies after the work is done ("Every new behavior has a corresponding test", "No tests were skipped"). HUMAN.md is the other half: what the human inspects during the run, at specific steps, with redirects. They complement each other: Verification is what the agent claims at the end; the gates are where the human samples the work early enough to steer it.

**Scope:** per-skill files in this repo. Not proposing an ecosystem standard.

**Maintenance:** each gate cites the stable prefix of a SKILL.md step heading (e.g. "Step 1: RED"), and `scripts/validate-skills.js` can gain a small check that every cited prefix matches a heading in the sibling SKILL.md. Prefix matching means cosmetic edits to a heading's tail don't break gates, while a removed or renumbered step fails CI instead of drifting silently. That catches structural drift only: if a step is rewritten in place, the gate can quietly stop making sense, which no machine check solves, and is one more reason to keep these files short and reviewed alongside any PR that touches their SKILL.md. The pilot is opt-in, so it adds no burden to contributors of other skills.

The evidence that shaped the format (it establishes the bottleneck and informed the design; the pilot is how we test whether this fixes it)

- Across 5,869 agent PRs matched with 5,869 human PRs ([MSR 2026](https://dl.acm.org/doi/10.1145/3793302.3793586)): humans intervene on 52.17% of agent PRs vs 83.59% of human ones, at roughly triple the median churn and double the duration; 58.02% of interventions are guidance vs 17.05% direct code fixes. Hence: few gates, early, guidance-shaped.
- The loop breaks from both ends: in [33,596 agentic PRs](https://arxiv.org/abs/2601.15195) the most common pattern behind rejected ones is the *reviewer* abandoning them (38% of the rejected sample), while [a second MSR 2026 study](https://arxiv.org/abs/2601.00753) finds *agents* ghosting PRs when feedback turns subjective (28.3% of agent PRs merge instantly with no negotiation). Hence: concrete, actionable redirects.
- Merge rates by task class run from documentation 84%, CI 79%, build 74% down to bug fix 64% and performance 55%, and doomed PRs grow large and multi-file ([same study](https://arxiv.org/abs/2601.15195)). Hence: gate count scaled by the skill's risk class, and diff growth as a stop rule.
- [Cognitive forcing functions reduce overreliance on AI](https://dl.acm.org/doi/10.1145/3449287), and the same study found people rate the heaviest friction least favorably. Hence: questions, capped at five minutes.
- In [METR's RCT](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/), developers 19% slower with AI believed they were 20% faster (METR flags the speed result as dated; the miscalibration is the durable lesson). Hence: objective stop rules, never "it feels productive".
- A recent [position paper (CMU, Stanford, Princeton)](https://arxiv.org/abs/2608.12355) argues coding-agent research is fixated on autonomy while the bottleneck shifts to human steering and verification.

Pre-flight: searched the catalog, open PRs, and the ecosystem; no existing convention covers this direction (closest: the `## Verification` sections, which are end-of-work exit criteria the agent self-checks; doubt-driven-development, which reviews decisions; and #523, which audits skill content).

Would a pilot of three files be welcome: test-driven-development, debugging-and-error-recovery, and shipping-and-launch? If the shape works we extend it; if not, it is three small files to delete. (It would also give #527's videos a natural script: a walkthrough is HUMAN.md acted out.)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.