addyosmani / addyosmani/agent-skills

Proposal: authoring rule against model-specific workarounds in skill content

Offen
#534 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
JavaScript
Sterne
93.8k
Forks
10k
Ø Merge
3 T. 15 Std.
Gemergte PRs (30 T.)
23

Beschreibung

## Summary

Add an authoring rule to `docs/skill-anatomy.md` (and a pre-flight line in `CONTRIBUTING.md`) that treats **model-specific workarounds in skill content as a portability liability**, not as helpful detail. A skill that exists to route around one model's execution failures can actively degrade a stronger model.

This repo currently has guidance on the portability of `references/` *paths* ([#361](https://github.com/addyosmani/agent-skills/issues/361)), but nothing on the portability of skill *content* across models.

## Evidence

Google Research recently published [WikiSkill](https://huggingface.co/papers/2608.27454) (arXiv:2608.27454). Most of it is about automated skill evolution and does not apply here, but its cross-model transfer study measures exactly the failure mode we keep seeing anecdotally.

Skills evolved by a small model, then run on a stronger one (Table 2, SpreadsheetBench):

| Inference model | Skill source | Score |
|---|---|---|
| Gemini-3.5-Flash | none | 50.5% |
| Gemini-3.5-Flash | Qwen-3.5-4B | **18.1%** |
| Gemini-3.5-Flash | Qwen-3.6-27B | 63.4% |

The skills authored against the weaker model cut the stronger model's score by roughly two thirds, well below using no skill at all. Their error analysis names two mechanisms:

1. The skills encode **low-level workarounds** (single-line Python commands, string-conversion rules) that help a weak model avoid execution failures but prevent a stronger model from writing comprehensive end-to-end scripts.
2. **Fragmented diagnostic procedures** introduce redundant tool calls that exhaust the interaction budget before the task completes.

The inverse also happens: skills evolved by Qwen-3.5-4B on OfficeQA slightly hurt that same model (30.2% to 28.5%) while lifting Qwen-3.6-27B from 42.1% to 52.9%. Discovering useful procedural knowledge and executing it are separable capabilities, so "it worked when I tried it" is weak evidence that a rule belongs in a shared catalog.

## Why this matters for this repo

Mechanism 2 is a plausible, and now named, explanation for [#423](https://github.com/addyosmani/agent-skills/issues/423) (the pack being slow on GPT-5.6): procedural fragmentation costs turns. Mechanism 1 gives a concrete authoring axis for the drift work in [#432](https://github.com/addyosmani/agent-skills/issues/432) and [#433](https://github.com/addyosmani/agent-skills/issues/433): rules written against a specific model generation are the ones most likely to age badly, and the failure is silent because they still read as reasonable advice.

## Proposed rule (draft wording)

> **Write the procedure, not the workaround.** A step that exists because one model gets a specific call wrong is a liability on every other model: it constrains agents that would have solved the task directly, and it spends turns that stronger models need for the actual work. If a rule cannot be justified without naming a model or a model version, it belongs in an issue, not in a skill.
>
> Two smells to check for before proposing skill content:
> - **Over-specified mechanics.** Prescribing an exact command form or serialization detail where the goal alone would do.
> - **Fragmented diagnostics.** A multi-step inspection sequence where one step would confirm the same thing. Every extra step is turns spent, and turn budget is finite.

## Scope

Docs only. No skill content changes in this issue; if the rule lands, auditing existing skills against it is separate follow-up work, and probably belongs with whatever [#432](https://github.com/addyosmani/agent-skills/issues/432) produces.

## Caveats, stated up front

The paper injects full skill content into the system prompt and says in its Limitations that this is deliberate, to remove skill triggering and retrieval as confounding variables. So none of its numbers say anything about description quality or routing, which is what our Tier 2 evals measure. The evaluated models are also mid-size open-weight models, not frontier coding agents. I am citing the *mechanism* and the direction of the effect, not proposing we treat the percentages as applying to this catalog.

@addyosmani @federicobartoli, does this look like a rule worth having written down, or is it already covered implicitly by the "no vague advice" bar? Related: #535 on keeping a durable record of rejected changes.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.