activeloopai / activeloopai/hivemind

skillopt: MetaStatus never advances past "proposed" — the meta-skill does dedup, not learning

Đang mở
#339 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
TypeScript
Star
1.6k
Fork
107
Merge trung bình
17 giờ 30 phút
Pull request đã merge (30 ngày)
6

Mô tả

`src/skillify/skillopt-meta.ts` documents the meta-skill as the optimizer's cross-run memory that should let the optimizer **learn which edits help**:

> ```
> // When the A/B gate lands, the recorded `status` (proposed → applied/reverted)
> // closes the loop so the optimizer learns which kinds of edits actually help
> // a given skill.
> ```

`MetaStatus` is declared as `"proposed" | "applied" | "reverted"` (line 15), but **only `"proposed"` is ever written** — `skillopt-meta.ts:82`. Nothing in `skillopt-worker.ts` or `skillopt-improve.ts` transitions an entry.

Consequences:
- `priorEditSummaries()` feeds the proposer every previously-tried edit **without any signal about whether it helped**, so the proposer can't prefer edit shapes that worked.
- The meta-skill's only working function today is **dedup** (`alreadyProposed` / fingerprint matching), which is real and valuable — but it's not what the header claims.
- The optimizer has **online** evaluation (every invocation is judged, including invocations of a version it just published) but no **attribution**: it never compares v_n vs v_n+1 on the same task, so it cannot say a given edit caused an improvement.

**Options:**
1. **Cheapest, no A/B needed:** on the next judged invocation of skill X at version v+1, write back `applied` (judge passed) or `reverted` (judge failed again) against the meta entry that produced v+1. That's a real outcome signal from data already being collected — it's sequential, not controlled, but it's honest and free.
2. Full A/B gate as originally designed (expensive — re-runs a user's task against two versions on their machine/token budget).
3. If neither is planned soon, narrow the type to `"proposed"` and update the header so the code stops advertising a capability it doesn't have.

Option 1 seems worth doing regardless; it turns `priorEditSummaries` from a blocklist into an actual prior.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.