Gate skill metadata by prompt relevance (as tools already do via virtualTools) so irrelevant installed skills cost zero tokens
- Dominant language
- TypeScript
- Stars
- 193k
- Forks
- 42.4k
- PR merge metrics
- PR metrics pending
Description
### Problem
Every installed skill's `name` + `description` is injected into the model prompt on
**every** turn, regardless of whether any skill could apply. Skills are gated only
*deterministically* (session type, feature flags, `disableModelInvocation`) and then the
full set is **listed** for the model to pick from. There is no per-query relevance gate on
*what metadata is surfaced at all*.
### The core problem: global install = silent per-turn tax on every session
I install a plugin or extension **once, globally** — and from then on **all** of its skills'
`name` + `description` are silently injected into the context of **every** prompt, in **every**
workspace, whether or not I care about them for the task at hand. I never opted into paying for
them per turn; installing the plugin did it for me (#323266: "installing VS Code extensions
silently adds more of them — extra agents, skills, tools … wasted tokens on things you don't use").
Concrete example: I install the MAUI plugin globally because I occasionally build a mobile app.
Now, while editing a **Blazor web app** in a completely unrelated repo, every prompt still carries
the token tax for a skill like `maui-shell-navigation` — which cannot possibly apply to what I'm
doing. The more plugins I install, the larger this fixed, always-on tax grows, and the worse
context rot gets — even though on any given prompt almost none of those skills are relevant.
This is not a "organize the catalog better" problem. Even a perfectly organized plugin imposes
this cost the moment it's installed globally, because there is no per-prompt gate deciding which
of its skills are worth surfacing.
### The mechanism already exists in this repo — for tools, not skills
VS Code already solves the identical problem for **tools** via `virtualTools/` (MIT,
`extensions/copilot/`):
- `virtualToolGrouper.ts` — `_getPredictedTools` / `_addEmbeddingMatchedTools` /
`recomputeEmbeddingRankings` embed the user query, cosine-rank it against a precomputed
tool-embedding cache, and place the top `NUM_EMBED_MATCHED_TOOLS` into an always-expanded
group ("Tools with high predicted relevancy for this query"); the rest are LLM-categorized
into activatable virtual groups once the count passes `START_GROUPING_AFTER_TOOL_COUNT`.
- `virtualToolsConstants.ts` — the tuning constants.
Skills get none of this. The asymmetry is the ask: **tools are relevance-ranked; skills are
dumped in full every turn.**
```mermaid
flowchart LR
Q[User prompt] --> T{Tools path}
Q --> S{Skills path}
T -->|embed + cosine rank| T1[Top-k relevant tools expanded
rest deferred into virtual groups]
S -->|deterministic gate only| S1[ALL skill name+desc listed
every turn, no relevance gate]
style T1 fill:#d6f5d6
style S1 fill:#f8d0d0
```
### Proposed solution — VirtualSkillGrouper (port the tool pipeline)
1. **Precompute** an embedding per skill from `name` + `description`; cache, recompute on change.
2. **Per turn**, embed the query; cosine-rank skills.
3. **Surface**: always-on "core" skills (deterministic, as today) + top-k above a similarity
floor. Everything else is **deferred** — omitted or collapsed behind a single
"more skills available (activate on demand)" affordance, mirroring virtual-group activation.
4. **Similarity floor is mandatory**: ToolRet (arXiv 2503.01763) shows generic embedders are
weak at tool/skill retrieval (nDCG@10 < 34) — never surface below threshold, and keep the
explicit `/skill-name` escape hatch (always reliable today).
### Eval plan (guardrail before shipping)
- **recall@k**: for a labeled set of (prompt → skill-that-should-fire), the surfaced set must
contain the correct skill ≥ target%. Safety metric — optimization must not silently drop a
needed skill.
- **tokens saved**: median skill-metadata tokens/turn, before vs after.
- **small-catalog no-op**: below a grouping threshold, byte-identical to today.
### Why now
Skill catalogs are growing via extensions, marketplace/plugin bundles, `~/.copilot/skills/`,
and `.github/skills/`. The reference implementation already exists in-repo for tools, so this
is a **port, not a research project.**
### Related
- **#323266** — same problem statement ("installing extensions silently adds skills … wasted
tokens"), but proposes *manual* on/off toggles; this issue is *automatic* relevance ranking.
- Also distinct from #315895 (reliability of skill *body* firing), #311874 (manual typeahead
picker), and #309641 (lazy-injecting *MCP tools* bundled with a skill).
Contributor guide
Assessment
This issue has not been assessed yet.