activeloopai / activeloopai/hivemind

skillopt: MetaStatus never advances past "proposed" — the meta-skill does dedup, not learning

Offen
#339 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
TypeScript
Sterne
1.6k
Forks
107
Ø Merge
17 Std. 30 Min.
Gemergte PRs (30 T.)
6

Beschreibung

`src/skillify/skillopt-meta.ts` documents the meta-skill as the optimizer's cross-run memory that should let the optimizer **learn which edits help**:

> ```
> // When the A/B gate lands, the recorded `status` (proposed → applied/reverted)
> // closes the loop so the optimizer learns which kinds of edits actually help
> // a given skill.
> ```

`MetaStatus` is declared as `"proposed" | "applied" | "reverted"` (line 15), but **only `"proposed"` is ever written** — `skillopt-meta.ts:82`. Nothing in `skillopt-worker.ts` or `skillopt-improve.ts` transitions an entry.

Consequences:
- `priorEditSummaries()` feeds the proposer every previously-tried edit **without any signal about whether it helped**, so the proposer can't prefer edit shapes that worked.
- The meta-skill's only working function today is **dedup** (`alreadyProposed` / fingerprint matching), which is real and valuable — but it's not what the header claims.
- The optimizer has **online** evaluation (every invocation is judged, including invocations of a version it just published) but no **attribution**: it never compares v_n vs v_n+1 on the same task, so it cannot say a given edit caused an improvement.

**Options:**
1. **Cheapest, no A/B needed:** on the next judged invocation of skill X at version v+1, write back `applied` (judge passed) or `reverted` (judge failed again) against the meta entry that produced v+1. That's a real outcome signal from data already being collected — it's sequential, not controlled, but it's honest and free.
2. Full A/B gate as originally designed (expensive — re-runs a user's task against two versions on their machine/token budget).
3. If neither is planned soon, narrow the type to `"proposed"` and update the header so the code stops advertising a capability it doesn't have.

Option 1 seems worth doing regardless; it turns `priorEditSummaries` from a blocklist into an actual prior.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.