addyosmani / addyosmani/agent-skills

Proposal: HUMAN.md, an evidence-based human protocol for each skill

オープン
#529 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
enhancement
主要言語
JavaScript
スター
94.1k
フォーク
10k
平均マージ
3日 15時間
マージ済み PR(30日)
23

説明

Every SKILL.md in this pack instructs the agent. Nothing instructs the human at the gates the skill itself creates. This proposes the counterpart file: HUMAN.md, next to SKILL.md, never loaded by the agent.

Here is what one would look like (test-driven-development, abridged):

## Gate 1: the failing test (after "Step 1: RED", ~2 min)
Read only the test diff, then the failure output. Ask:
- Does it fail for the right reason? (read the error message, not the code)
- Would a trivially wrong implementation pass it?
→ redirect: "Strengthen the assertion, it currently accepts X."
- Behavior, or implementation detail?
→ redirect: "Rewrite the test against the public interface only."

## Gate 2: after the refactor, before commit (after "Step 3: REFACTOR", ~3 min)
Read the implementation diff only. Ask:
- Did anything change outside the slice?
→ redirect: "Revert changes outside the task scope."
- Do the test names still describe behavior after the refactor?

## Stop rules
- Third failed iteration on the same test → write the test yourself.
- Diff grew beyond the files named in the plan → pause and re-scope.

That is the whole format: gates that cite SKILL.md step headings, gate count scaled by how often that skill's class of work fails, a review phrased as questions (~5 min total per run), redirects the agent can act on, objective stop rules.

**Why a separate file, not a `## For humans` section:** SKILL.md is loaded into agent context in full, so a human section would cost tokens on every invocation and put the reviewer's checklist inside the model's prompt. A separate file costs zero tokens, and since no harness parses it, it is tool-agnostic by construction.

**Why not the existing `## Verification` sections:** every skill already ends with one, but it is a list of exit criteria the agent self-certifies after the work is done ("Every new behavior has a corresponding test", "No tests were skipped"). HUMAN.md is the other half: what the human inspects during the run, at specific steps, with redirects. They complement each other: Verification is what the agent claims at the end; the gates are where the human samples the work early enough to steer it.

**Scope:** per-skill files in this repo. Not proposing an ecosystem standard.

**Maintenance:** each gate cites the stable prefix of a SKILL.md step heading (e.g. "Step 1: RED"), and `scripts/validate-skills.js` can gain a small check that every cited prefix matches a heading in the sibling SKILL.md. Prefix matching means cosmetic edits to a heading's tail don't break gates, while a removed or renumbered step fails CI instead of drifting silently. That catches structural drift only: if a step is rewritten in place, the gate can quietly stop making sense, which no machine check solves, and is one more reason to keep these files short and reviewed alongside any PR that touches their SKILL.md. The pilot is opt-in, so it adds no burden to contributors of other skills.

The evidence that shaped the format (it establishes the bottleneck and informed the design; the pilot is how we test whether this fixes it)

- Across 5,869 agent PRs matched with 5,869 human PRs ([MSR 2026](https://dl.acm.org/doi/10.1145/3793302.3793586)): humans intervene on 52.17% of agent PRs vs 83.59% of human ones, at roughly triple the median churn and double the duration; 58.02% of interventions are guidance vs 17.05% direct code fixes. Hence: few gates, early, guidance-shaped.
- The loop breaks from both ends: in [33,596 agentic PRs](https://arxiv.org/abs/2601.15195) the most common pattern behind rejected ones is the *reviewer* abandoning them (38% of the rejected sample), while [a second MSR 2026 study](https://arxiv.org/abs/2601.00753) finds *agents* ghosting PRs when feedback turns subjective (28.3% of agent PRs merge instantly with no negotiation). Hence: concrete, actionable redirects.
- Merge rates by task class run from documentation 84%, CI 79%, build 74% down to bug fix 64% and performance 55%, and doomed PRs grow large and multi-file ([same study](https://arxiv.org/abs/2601.15195)). Hence: gate count scaled by the skill's risk class, and diff growth as a stop rule.
- [Cognitive forcing functions reduce overreliance on AI](https://dl.acm.org/doi/10.1145/3449287), and the same study found people rate the heaviest friction least favorably. Hence: questions, capped at five minutes.
- In [METR's RCT](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/), developers 19% slower with AI believed they were 20% faster (METR flags the speed result as dated; the miscalibration is the durable lesson). Hence: objective stop rules, never "it feels productive".
- A recent [position paper (CMU, Stanford, Princeton)](https://arxiv.org/abs/2608.12355) argues coding-agent research is fixated on autonomy while the bottleneck shifts to human steering and verification.

Pre-flight: searched the catalog, open PRs, and the ecosystem; no existing convention covers this direction (closest: the `## Verification` sections, which are end-of-work exit criteria the agent self-checks; doubt-driven-development, which reviews decisions; and #523, which audits skill content).

Would a pilot of three files be welcome: test-driven-development, debugging-and-error-recovery, and shipping-and-launch? If the shape works we extend it; if not, it is three small files to delete. (It would also give #527's videos a natural script: a walkthrough is HUMAN.md acted out.)

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start from one or two existing SKILL.md files to mirror their section structure and define sibling HUMAN.md files that reference the stable step prefixes. Then inspect scripts/validate-skills.js to understand how skills are loaded and where a prefix check against SKILL.md headings would be added. Implement the pilot by adding HUMAN.md for the three listed skills and run the repository’s skills validation flow to verify it still passes. Done when the three files are in place, headings match SKILL.md step prefixes, and validation has no regressions.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
javascript
領域
documentation
issue の種類
ドキュメント
難易度
3/5
見積もり時間
1〜2日
活発さ
活発
明瞭さ
おおむね明確
初心者へのやさしさ
70/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。