Automate plugin skill sync from .github/skills/ to plugins/azure-sdk-tools/skills/
- Dominant language
- C#
- Stars
- 135
- Forks
- 260
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 143
Description
### Summary
Skill content lives in two places that must be kept in sync by hand:
- **`.github/skills//`** — agent skills for this repo
- **`plugins/azure-sdk-tools/skills//`** — skills bundled into the Copilot CLI plugin published to the `Azure/azure-sdk-tools` marketplace
Every skill change has to be applied to both. In practice, one half drifts. Recent example: PR #15956 improved GEPA scores under `.github/skills/`; the `plugins/` copies are still untouched.
The Copilot CLI marketplace pulls straight from this repo at `HEAD` (`copilot plugin marketplace add Azure/azure-sdk-tools` + `plugins/azure-sdk-tools/.claude-plugin/plugin.json` pointing at `./skills/`), so whatever is merged to `main` IS what gets published. There's no separate publish pipeline to hook into — the automation has to be a CI workflow that runs on push to `main`.
### Findings from a sync attempt
A naive "treat plugin tree as generated, `Copy-Item -Recurse .github/skills/* plugins/.../skills/`" approach breaks the plugin. The two trees aren't a 1:1 copy:
| Aspect | `.github/skills/` | `plugins/azure-sdk-tools/skills/` |
|---|---|---|
| Directory naming | `azsdk-common-prepare-release-plan` | `prepare-release-plan` |
| `name:` in frontmatter | `azsdk-common-prepare-release-plan` | `prepare-release-plan` |
| Skill set | has `azsdk-common-api-review` (not in plugin) | has `live-and-recorded-tests` (not in source) |
| Aux files | `references/` only | adds `evals/`, `tasks/`, `trigger_tests.yaml` etc. |
A raw recursive copy would:
- rename every plugin skill (breaking its identity in the marketplace),
- wipe the plugin's `evals/` and `tasks/` directories,
- ship a skill (`api-review`) the plugin team may not want to publish,
- delete `live-and-recorded-tests` entirely.
### End-state
The two trees stay in sync via a workflow that runs on every push to `main`. Contributors only ever edit `.github/skills/`. The plugin tree is still committed to the repo (because the marketplace serves directly from `HEAD`), but it's *maintained by the workflow*, not by humans.
### End-to-end flow
```mermaid
flowchart TD
A["Author opens a PR
(edits .github/skills/foo/SKILL.md)"] --> B["Optional PR comment:
'FYI — merging this will update N plugin file(s)'"]
B --> C["PR reviewed + merged to main"]
C --> D["push-to-main workflow fires"]
D --> E["Run sync transform
(reads skill-sync-map.json)"]
E --> F{"plugin tree
changed?"}
F -->|no| G["Workflow exits clean"]
F -->|yes| H["Bot commits sync result
to main with [skip ci]"]
H --> I["Marketplace serves
updated plugin"]
```
### Building blocks
1. **Declarative mapping** (`plugins/azure-sdk-tools/skill-sync-map.json`)
Per skill: source path under `.github/skills/`, destination path under `plugins/azure-sdk-tools/skills/`, list of sub-paths to copy (default `SKILL.md` + `references/`), optional `nameOverride` for the frontmatter `name:` field. Plugin-only assets (`evals/`, `tasks/`, etc.) are simply not listed and stay untouched.
2. **Sync transform** (single script, e.g. `eng/scripts/sync-plugin-skills.ps1`)
Reads the mapping, regenerates the destination subtree, rewrites the `name:` frontmatter where needed. Idempotent. Works on a contributor's laptop or a CI agent — same command, same output.
3. **Auto-sync workflow** (`.github/workflows/plugin-skill-sync.yml`, `on: push: branches: [main]`, paths-filtered to `.github/skills/**` and the map)
Runs the transform on a clean checkout of `main`. If the working tree changed, commits as the bot identity and pushes back to `main` with `[skip ci]` in the message so it doesn't loop. Permissions: `contents: write`.
### What the auto-commit looks like in history
Two commits per skill change:
- `Improve GEPA score for prepare-release-plan` (human, on the PR)
- `[skill-sync-bot]: Sync plugins from .github/skills` (bot, post-merge to `main`)
Easy to filter (`git log --author=skill-sync-bot`), easy to revert (revert both commits together).
### Trade-offs
| Aspect | This proposal | PR-gate alternative |
|---|---|---|
| Contributor friction | Zero — edit one file, merge, done | Author must re-run a command if the gate fails |
| Reviewer visibility | Reviewers see only the human edit. Optional FYI comment closes that gap | PR-gate shows the propagated change in the same PR |
| Rollback story | Revert two commits (human + bot) | Revert one commit |
| Permissions | Workflow needs `contents: write` on `main` | No write access needed |
| External fork PRs | Works trivially — sync runs after squash-merge | Auto-commit-to-PR-branch can't write to fork branches |
| Failure modes | Sync script bug ships to `main` before review; mitigation is fast revert + the FYI comment | None new |
| Marketplace consumers | See updated plugin within a minute of merge | Same |
### Out of scope
- Renaming `.github/skills/azsdk-common-*` directories to drop the prefix (would simplify the map but breaks every existing reference and skill-invocation grader; separate decision).
- Sync to `azure-rest-api-specs` repo — tracked in #15797.
### Sequencing
1. Land the mapping + transform + baseline sync as one PR. No workflows yet; just proves the transform is correct against the current trees.
2. Land the `push: main` auto-sync workflow as a follow-up PR. Branch protection / bot identity decisions resolved here.
3. (Optional) Land the PR-time FYI comment workflow.
Three small PRs, each independently reviewable and revertible.
### Related
- PR #15956 — skill quality improvements that need to be mirrored.
- #15797 — tools→specs repo sync (different problem).
- #15950 — skill ownership gap that surfaced this.
Contributor guide
Assessment
This issue has not been assessed yet.