anthropics / anthropics/skills

aggregate_benchmark.py and eval-viewer/generate_review.py disagree on where grading.json lives for the documented single-run layout

オープン
#1,425 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
176k
フォーク
20.9k
平均マージ
7時間 21分
マージ済み PR(30日)
5

説明

`skills/skill-creator/scripts/aggregate_benchmark.py` and `skills/skill-creator/eval-viewer/generate_review.py` accept different directory layouts for `grading.json`, and the mismatch causes the benchmark script to silently report 0% for every stat with no error at all.

**The mismatch**

- `generate_review.py`'s `build_run()` (eval-viewer/generate_review.py:130-138) checks `/grading.json` directly (or `/grading.json`) — no `run-N/` subfolder required.
- `aggregate_benchmark.py`'s `load_run_results()` (scripts/aggregate_benchmark.py:100-106) requires a `run-N/` subfolder to even recognize a config directory as valid (`if not list(config_dir.glob("run-*")): continue`). If `grading.json` sits directly in `/`, the whole config directory is skipped.

SKILL.md's own Step 4 documents saving a single run's grading straight to `grading.json` in the run directory — following that literally satisfies the viewer but silently breaks the benchmark aggregation.

**Why it's silent:** the "Warning: grading.json not found" line (aggregate_benchmark.py:116) lives inside the loop over `run-*` subdirectories. For the flat layout that loop never executes, so no warning prints at all — `benchmark.json`/`benchmark.md` just come back with every stat at 0, which reads as "the skill scored zero" rather than "no data was found."

**Repro**

```
mkdir -p ws/eval-0/with_skill/outputs
echo '{"summary":{"pass_rate":1.0,"passed":3,"failed":0,"total":3},"expectations":[]}' > ws/eval-0/with_skill/grading.json
python -m scripts.aggregate_benchmark ws --skill-name test
cat ws/benchmark.md # Pass Rate: 0% ± 0% — should be 100%
```

**Suggested fix**

`load_run_results` should fall back to treating `/` itself as run 1 when no `run-N/` subfolder exists but `grading.json` sits directly inside it — mirroring what the viewer already accepts. As a cheap defense-in-depth addition, `aggregate_benchmark.py` should also print a visible error when zero `grading.json` files are found at all, rather than silently returning all-zero stats.

I've verified a minimal patch along these lines works correctly across all three documented layouts (flat single-run, `run-N/`, and the legacy `runs/` wrapper) with no regressions, and I'm happy to open a PR with it if that's useful.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

Start with skills/skill-creator/scripts/aggregate_benchmark.py, especially load_run_results(), and compare its layout handling with build_run() in skills/skill-creator/eval-viewer/generate_review.py. Run the documented flat-layout reproduction, then verify aggregation works for flat, run-N/, and legacy runs/ layouts and reports a visible error when no grading.json files are found.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
testing, tooling
issue の種類
バグ
難易度
3/5
見積もり時間
1〜2日
活発さ
静か
明瞭さ
明確に書かれている
初心者へのやさしさ
76/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。