microsoft / microsoft/Dataverse-skills

Token-budget headroom: split near-cap skill bodies into references (EVAL-BUDGET-02)

Open
#111 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
226
Forks
61
Avg merge
3d 9h
Merged PRs (30d)
7

Description

## Problem

Skill `SKILL.md` **bodies** are hard-capped at **5,000 tokens** (EVAL-BUDGET-02 in `.github/evals/static_checks.py`, per Anthropic's Level-2 skills-loading model). Four skills are at/near the cap, so any new content requires offsetting trims and risks a red build:

| Skill | Body tokens | Headroom |
|---|---|---|
| dv-metadata | 5000 | 0 |
| dv-overview | 5000 | 0 |
| dv-admin | 4981 | 19 |
| dv-connect | 4969 | 31 |

(Measured 2026-08-09; the other four skills have ~900-3000 tokens of headroom.)

## Why it matters

- Adding even a one-line note to these skills fails `static_checks` (EVAL-BUDGET-02) unless something is trimmed first -- hit repeatedly during recent auth work.
- It discourages documenting real behavior in the most-used skills, and makes small edits disproportionately expensive.

## Proposed approach

Move detail into `references/.md` (Level 3, unlimited) with a pointer left in the body -- the sanctioned pattern (CLAUDE.md: "When a skill body grows past ~4,000 tokens, split long content into `references/`"). Per skill:

- **dv-metadata (5000):** move long form / view / relationship / option-set detail into a `references/` topic (e.g. `references/forms-views.md`); keep a Quick Reference + pointer in the body.
- **dv-connect (4969):** move Step 6/7 MCP config/verification detail into the existing `references/mcp-configuration.md`; keep the step skeleton + pointer.
- **dv-overview (5000):** **exempt from the references-split nudge** per the authoring standard (it is the routing hub, loaded as one surface) -- relieve by tightening prose and moving the SDK method cheat-sheet into a small reference.
- **dv-admin (4981):** if more headroom is needed, the 37-toggle allowlist detail can move to a reference.

Target ~10-15% body-token headroom per near-cap skill.

## How to measure

`static_checks` only reports on failure, so measure directly with the cl100k encoder (matches CAT-8):

```python
import re, tiktoken
from pathlib import Path
enc = tiktoken.get_encoding("cl100k_base")
for md in sorted(Path(".github/plugins/dataverse/skills").glob("*/SKILL.md")):
text = md.read_text(encoding="utf-8")
body = re.sub(r"^---\n.*?\n---\n?", "", text, count=1, flags=re.DOTALL)
print(f"{md.parent.name:<14}{len(enc.encode(body)):>6}")
```

## Acceptance criteria

- Each near-cap skill (dv-metadata, dv-overview, dv-connect, dv-admin) has >= ~10% body-token headroom.
- No routing triggers or taught behavior lost (moved content is linked from the body).
- `python .github/evals/static_checks.py` passes.
- `python .github/evals/version_bump_check.py --base origin/main` passes (PATCH bump -- refactor only, no skill added or removed).

## Context

Surfaced during the host-aware auth work (#110). Deliberately kept out of that PR for one-net-outcome scope discipline.

Contributor guide

Open the contributing guide

Research direction

Read the four skill bodies under .github/plugins/dataverse/skills/*/SKILL.md and the authoring guidance in CLAUDE.md, then run the cl100k measurement script from the issue. Move the specified detail into references while preserving routing and taught behavior, and verify the target headroom. Finish by running .github/evals/static_checks.py and .github/evals/version_bump_check.py --base origin/main.

Written by the indexing model from the issue text.

Assessment

Tech stack
markdown, python
Domain
documentation, tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.