MetaMask / MetaMask/skills

Automate the context-budget report on skill PRs, and gate on it

Open
#96 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
23
Forks
11
Avg merge
3d 8h
Merged PRs (30d)
20

Description

**Size:** M | **Files:** `.github/workflows/`, `.github/scripts/`, `tools/skill-schema.mjs`

---

## Problem

A reviewer cannot read a skills PR. The nine currently open add **68 files, ~90,000 words, ~112k tokens** — roughly six hours of sustained reading before any judgement. In practice that means skills merge on the strength of their PR description.

The part a reviewer most needs is also the part hardest to see by reading: what the skill *costs*. Measured by hand across the 26 new skills:

| Tier | When paid | Median | Max |
|---|---|---|---|
| Frontmatter | every agent, every run, whether or not the skill is used | — | **12,837 chars total** (~3,209 tok) |
| Selected | agent opens the skill | ~1,613 tok | 9,059 tok |
| + references & knowledge | agent follows the pointers | ~3,511 tok | 35,011 tok |

Frontmatter is the tier that matters and the one nobody sees: it is paid by every agent on every run once installed, including runs that never touch the skill, and it is the only tier that cannot be opted out of after install.

The same measurement found **five of 26 skills over the 1,024-character description ceiling** that #47 enforces — each of which would have failed CI on merge, discovered by measurement rather than by review.

None of this is visible in a diff, and none of it is currently computed.

---

## Solution

**1. Post the report automatically.** On any PR touching `domains/**`, have `metamaskbot` comment a context-budget table: per skill, its frontmatter chars, selected tokens, and ceiling with references followed — plus how those compare to the catalogue median. Idempotent upsert on re-push, so the comment tracks the head rather than accumulating.

Measurement has to run against an **actual install** (`tools/install` into a temp target), not source bytes. Source-byte sums are wrong in two ways found while doing this by hand: they miss the repo-overlay merge, and they double-count domain knowledge, which is copied per skill but read once per run.

Use `main`'s installer rather than the PR branch's, or a branch behind `main` reports its knowledge as undelivered when on merge it ships.

**2. Gate on the numbers.** Fail CI where a threshold is objective, warn where it is a judgement call:

- **Fail** — any `description` over `DESCRIPTION_MAX`. Already enforced by #47 for changed skills; this extends the same number into the budget report so the two cannot disagree.
- **Warn** — a skill whose selected cost exceeds some multiple of the catalogue median. `pr-validate` was 5.6× before being restructured, which was worth knowing at review time rather than after merge.
- **Warn** — total always-on growth in a single PR above a threshold, since that is the tier every consumer pays forever.

Thresholds belong in `tools/skill-schema.mjs` beside `DESCRIPTION_MAX`, not in the workflow, so they are reviewable and testable like the rest of the schema.

---

## Acceptance Criteria

- [ ] A PR touching `domains/**` receives a `metamaskbot` comment with per-skill frontmatter / selected / ceiling figures
- [ ] The comment upserts on re-push rather than posting a second one
- [ ] Measurement runs against a real install, using `main`'s installer, not source bytes
- [ ] Knowledge copied per skill is counted once per run, not once per copy
- [ ] A description over `DESCRIPTION_MAX` fails CI, with the same number `lint-skill-entry` uses
- [ ] Cost thresholds live in `tools/skill-schema.mjs` and are covered by tests
- [ ] The check runs on changed skills only, so pre-existing catalogue weight never blocks an unrelated PR
- [ ] Verified to fire: a skill authored over each threshold fails or warns as specified

---

## Notes

Deliberately **cost, not value**. A 35k-token skill that saves an hour is a bargain, and nothing here says which those are — it gives a reviewer one measured input, not a verdict. Judging whether the guidance is *correct* still needs a domain expert, or an A/B of agent output with and without the skill, which is a separate and much larger piece of work.

Companion to #47 (structural validation) and #87 (reference resolution): those check that a skill is well-formed and that what it points at exists. This checks what it costs.

Contributor guide

Open the contributing guide

Research direction

Start by reading the workflows and scripts under .github/workflows/ and .github/scripts/, then inspect tools/install, tools/skill-schema.mjs, lint-skill-entry, and pr-validate. Trace how a real temporary install can measure changed skills and how the existing description ceiling is enforced. Done means an idempotent metamaskbot report, objective CI gating with thresholds in the schema, and tests covering threshold behavior and changed skills only.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, javascript
Domain
ci-cd, devops, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.