activeloopai / activeloopai/hivemind

skillopt: MetaStatus never advances past "proposed" — the meta-skill does dedup, not learning

Abierto
#339 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
TypeScript
Estrellas
1.6k
Forks
107
Merge medio
17 h 30 min
PR fusionados (30 d)
6

Descripción

`src/skillify/skillopt-meta.ts` documents the meta-skill as the optimizer's cross-run memory that should let the optimizer **learn which edits help**:

> ```
> // When the A/B gate lands, the recorded `status` (proposed → applied/reverted)
> // closes the loop so the optimizer learns which kinds of edits actually help
> // a given skill.
> ```

`MetaStatus` is declared as `"proposed" | "applied" | "reverted"` (line 15), but **only `"proposed"` is ever written** — `skillopt-meta.ts:82`. Nothing in `skillopt-worker.ts` or `skillopt-improve.ts` transitions an entry.

Consequences:
- `priorEditSummaries()` feeds the proposer every previously-tried edit **without any signal about whether it helped**, so the proposer can't prefer edit shapes that worked.
- The meta-skill's only working function today is **dedup** (`alreadyProposed` / fingerprint matching), which is real and valuable — but it's not what the header claims.
- The optimizer has **online** evaluation (every invocation is judged, including invocations of a version it just published) but no **attribution**: it never compares v_n vs v_n+1 on the same task, so it cannot say a given edit caused an improvement.

**Options:**
1. **Cheapest, no A/B needed:** on the next judged invocation of skill X at version v+1, write back `applied` (judge passed) or `reverted` (judge failed again) against the meta entry that produced v+1. That's a real outcome signal from data already being collected — it's sequential, not controlled, but it's honest and free.
2. Full A/B gate as originally designed (expensive — re-runs a user's task against two versions on their machine/token budget).
3. If neither is planned soon, narrow the type to `"proposed"` and update the header so the code stops advertising a capability it doesn't have.

Option 1 seems worth doing regardless; it turns `priorEditSummaries` from a blocklist into an actual prior.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.