activeloopai / activeloopai/hivemind
skillopt: MetaStatus never advances past "proposed" — the meta-skill does dedup, not learning
- Vorherrschende Sprache
- TypeScript
- Sterne
- 1.6k
- Forks
- 107
- Ø Merge
- 17 Std. 30 Min.
- Gemergte PRs (30 T.)
- 6
Beschreibung
`src/skillify/skillopt-meta.ts` documents the meta-skill as the optimizer's cross-run memory that should let the optimizer **learn which edits help**:
> ```
> // When the A/B gate lands, the recorded `status` (proposed → applied/reverted)
> // closes the loop so the optimizer learns which kinds of edits actually help
> // a given skill.
> ```
`MetaStatus` is declared as `"proposed" | "applied" | "reverted"` (line 15), but **only `"proposed"` is ever written** — `skillopt-meta.ts:82`. Nothing in `skillopt-worker.ts` or `skillopt-improve.ts` transitions an entry.
Consequences:
- `priorEditSummaries()` feeds the proposer every previously-tried edit **without any signal about whether it helped**, so the proposer can't prefer edit shapes that worked.
- The meta-skill's only working function today is **dedup** (`alreadyProposed` / fingerprint matching), which is real and valuable — but it's not what the header claims.
- The optimizer has **online** evaluation (every invocation is judged, including invocations of a version it just published) but no **attribution**: it never compares v_n vs v_n+1 on the same task, so it cannot say a given edit caused an improvement.
**Options:**
1. **Cheapest, no A/B needed:** on the next judged invocation of skill X at version v+1, write back `applied` (judge passed) or `reverted` (judge failed again) against the meta entry that produced v+1. That's a real outcome signal from data already being collected — it's sequential, not controlled, but it's honest and free.
2. Full A/B gate as originally designed (expensive — re-runs a user's task against two versions on their machine/token budget).
3. If neither is planned soon, narrow the type to `"proposed"` and update the header so the code stops advertising a capability it doesn't have.
Option 1 seems worth doing regardless; it turns `priorEditSummaries` from a blocklist into an actual prior.
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.