flyteorg / flyteorg/flyte

Skill Forge — Flyte-native writer/tester loop to harden deploy skills

Open
#7,861 1 comment 0 reactions 1 assignee Claimed by @mhotan View on GitHub
Dominant language
Go
Stars
7.5k
Forks
886
Avg merge
1d 12h
Merged PRs (30d)
120

Description

A bounded Flyte v2 workflow where a **writer** agent edits a deploy `SKILL.md` and an independent **tester** agent stands the deployment up from scratch and runs `flyte-functional-tests`, looping writer↔tester until the skill is green (or a cap is hit), then emitting a report and opening PRs. Dogfoods Flyte to build Flyte deploy skills.

**Design plan (artifact):** [https://claude.ai/code/artifact/558e46f9-3019-4726-be7b-f29d0c372c63]()

### The loop

```
writer (Agent A, strong model) → edits SKILL.md
→ tester (Agent B, different vendor) → deploys for real, unstubbed CLIs
→ gate: flyte-functional-tests (deterministic pass/fail)
→ fail: verdict feeds back to writer (≤ N iterations)
→ green/improved: report + open PR
```

### Key properties

* Both are real `flyte.ai.agents.Agent` instances; writer never deploys, tester never edits.
* **Reusable container** keeps Docker/k3d + image cache warm across passes.
* Tester independence: different model/vendor grades a fresh deploy blind.
* Bounds: iteration cap + budget cap + no-improvement early stop; teardown every pass; PR-never-merge.

### First target

`deploy-flyte-k3d` — hermetic (in-cluster PostgreSQL + RustFS on k3d), no cloud creds. Cloud targets later via the applied `flyte-agent-plugins-ci` keyless identities.

### Phases

- [ ] **P1 — Scaffold the loop.** `evals/forge/forge_wf.py`: bounded `@env.task` loop + `MemoryStore`, writer/tester Agent builders, Verdict schema. Stubbed tester to prove control flow + reporting.
- [ ] **P2 — Real tester on k3d.** DinD reusable container; run `deploy-flyte-k3d` for real; verify with `flyte-functional-tests`. Proves the loop hermetically.
- [ ] **P3 — Report & PR tasks.** HTML scorecard, markdown artifact, `open_pr` (fork push, cross-fork PR, perennial fast-forward).
- [ ] **P4 — Matrix fan-out.** `flyte.map` across the skill matrix; cloud targets via the isolated identities; nightly cron.
- [ ] **P5 — Independence & judging polish.** Distinct writer/tester vendors, no-improvement early stop, budget-aware iteration, per-skill rating trend.

Full design + rationale in the artifact.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.