activeloopai / activeloopai/hivemind

skillopt: MetaStatus never advances past "proposed" — the meta-skill does dedup, not learning

Ouverte
#339 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
TypeScript
Étoiles
1.6k
Forks
107
Merge moyen
17 h 30 min
PR mergées (30 j)
6

Description

`src/skillify/skillopt-meta.ts` documents the meta-skill as the optimizer's cross-run memory that should let the optimizer **learn which edits help**:

> ```
> // When the A/B gate lands, the recorded `status` (proposed → applied/reverted)
> // closes the loop so the optimizer learns which kinds of edits actually help
> // a given skill.
> ```

`MetaStatus` is declared as `"proposed" | "applied" | "reverted"` (line 15), but **only `"proposed"` is ever written** — `skillopt-meta.ts:82`. Nothing in `skillopt-worker.ts` or `skillopt-improve.ts` transitions an entry.

Consequences:
- `priorEditSummaries()` feeds the proposer every previously-tried edit **without any signal about whether it helped**, so the proposer can't prefer edit shapes that worked.
- The meta-skill's only working function today is **dedup** (`alreadyProposed` / fingerprint matching), which is real and valuable — but it's not what the header claims.
- The optimizer has **online** evaluation (every invocation is judged, including invocations of a version it just published) but no **attribution**: it never compares v_n vs v_n+1 on the same task, so it cannot say a given edit caused an improvement.

**Options:**
1. **Cheapest, no A/B needed:** on the next judged invocation of skill X at version v+1, write back `applied` (judge passed) or `reverted` (judge failed again) against the meta entry that produced v+1. That's a real outcome signal from data already being collected — it's sequential, not controlled, but it's honest and free.
2. Full A/B gate as originally designed (expensive — re-runs a user's task against two versions on their machine/token budget).
3. If neither is planned soon, narrow the type to `"proposed"` and update the header so the code stops advertising a capability it doesn't have.

Option 1 seems worth doing regardless; it turns `priorEditSummaries` from a blocklist into an actual prior.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.