Skill Forge — Flyte-native writer/tester loop to harden deploy skills
- Dominant language
- Go
- Stars
- 7.5k
- Forks
- 886
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 120
Description
A bounded Flyte v2 workflow where a **writer** agent edits a deploy `SKILL.md` and an independent **tester** agent stands the deployment up from scratch and runs `flyte-functional-tests`, looping writer↔tester until the skill is green (or a cap is hit), then emitting a report and opening PRs. Dogfoods Flyte to build Flyte deploy skills.
**Design plan (artifact):** [https://claude.ai/code/artifact/558e46f9-3019-4726-be7b-f29d0c372c63]()
### The loop
```
writer (Agent A, strong model) → edits SKILL.md
→ tester (Agent B, different vendor) → deploys for real, unstubbed CLIs
→ gate: flyte-functional-tests (deterministic pass/fail)
→ fail: verdict feeds back to writer (≤ N iterations)
→ green/improved: report + open PR
```
### Key properties
* Both are real `flyte.ai.agents.Agent` instances; writer never deploys, tester never edits.
* **Reusable container** keeps Docker/k3d + image cache warm across passes.
* Tester independence: different model/vendor grades a fresh deploy blind.
* Bounds: iteration cap + budget cap + no-improvement early stop; teardown every pass; PR-never-merge.
### First target
`deploy-flyte-k3d` — hermetic (in-cluster PostgreSQL + RustFS on k3d), no cloud creds. Cloud targets later via the applied `flyte-agent-plugins-ci` keyless identities.
### Phases
- [ ] **P1 — Scaffold the loop.** `evals/forge/forge_wf.py`: bounded `@env.task` loop + `MemoryStore`, writer/tester Agent builders, Verdict schema. Stubbed tester to prove control flow + reporting.
- [ ] **P2 — Real tester on k3d.** DinD reusable container; run `deploy-flyte-k3d` for real; verify with `flyte-functional-tests`. Proves the loop hermetically.
- [ ] **P3 — Report & PR tasks.** HTML scorecard, markdown artifact, `open_pr` (fork push, cross-fork PR, perennial fast-forward).
- [ ] **P4 — Matrix fan-out.** `flyte.map` across the skill matrix; cloud targets via the isolated identities; nightly cron.
- [ ] **P5 — Independence & judging polish.** Distinct writer/tester vendors, no-improvement early stop, budget-aware iteration, per-skill rating trend.
Full design + rationale in the artifact.
Contributor guide
Assessment
This issue has not been assessed yet.