addyosmani / addyosmani/agent-skills

Drift data for 3 of these skills across the sonnet-4-6 → sonnet-5 release

Đang mở
#432 6 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
JavaScript
Star
93.8k
Fork
10k
Merge trung bình
3 ngày 15 giờ
Pull request đã merge (30 ngày)
23

Mô tả

Hi Addy — I run a small OSS project (Driftproof) that re-tests agent skills across model releases: same suite, with/without the skill, judge sampled 5x so every score carries a confidence band, and a regression is only claimed when bands separate AND the move clears a 0.05 effect floor.

For our first report I wrote eval suites for three skills from this repo (code-review-and-quality, git-workflow-and-versioning, documentation-and-adrs), each case grounded in a claim the skill's own SKILL.md makes, and ran them across the sonnet-4-6 → sonnet-5 release. Results, with receipts:

- code-review-and-quality: WITHIN NOISE (held up)
- git-workflow-and-versioning: 1 case regressed (semver-clean-bump)
- documentation-and-adrs: MIXED (one sharp regression, one improvement)

Full tables + methodology: https://driftproofhq.com/reports/001/
The suites themselves: https://github.com/driftproofhq/driftproof/tree/main/suites

Offer, take or leave any of it: the suites are Apache-2.0 and yours to adopt upstream if useful; `npx driftproof run` re-runs them on any model with your own key; and there's a GitHub Action + badge if continuous verification on releases is interesting. If you think any case tests something your skill doesn't actually claim, I'd genuinely like to know — suite fairness is the part I care most about getting right.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.