anthropics / anthropics/skills

aggregate_benchmark.py and eval-viewer/generate_review.py disagree on where grading.json lives for the documented single-run layout

Open
#1,425 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
176k
Forks
20.8k
Avg merge
7h 21m
Merged PRs (30d)
5

Description

`skills/skill-creator/scripts/aggregate_benchmark.py` and `skills/skill-creator/eval-viewer/generate_review.py` accept different directory layouts for `grading.json`, and the mismatch causes the benchmark script to silently report 0% for every stat with no error at all.

**The mismatch**

- `generate_review.py`'s `build_run()` (eval-viewer/generate_review.py:130-138) checks `/grading.json` directly (or `/grading.json`) — no `run-N/` subfolder required.
- `aggregate_benchmark.py`'s `load_run_results()` (scripts/aggregate_benchmark.py:100-106) requires a `run-N/` subfolder to even recognize a config directory as valid (`if not list(config_dir.glob("run-*")): continue`). If `grading.json` sits directly in `/`, the whole config directory is skipped.

SKILL.md's own Step 4 documents saving a single run's grading straight to `grading.json` in the run directory — following that literally satisfies the viewer but silently breaks the benchmark aggregation.

**Why it's silent:** the "Warning: grading.json not found" line (aggregate_benchmark.py:116) lives inside the loop over `run-*` subdirectories. For the flat layout that loop never executes, so no warning prints at all — `benchmark.json`/`benchmark.md` just come back with every stat at 0, which reads as "the skill scored zero" rather than "no data was found."

**Repro**

```
mkdir -p ws/eval-0/with_skill/outputs
echo '{"summary":{"pass_rate":1.0,"passed":3,"failed":0,"total":3},"expectations":[]}' > ws/eval-0/with_skill/grading.json
python -m scripts.aggregate_benchmark ws --skill-name test
cat ws/benchmark.md # Pass Rate: 0% ± 0% — should be 100%
```

**Suggested fix**

`load_run_results` should fall back to treating `/` itself as run 1 when no `run-N/` subfolder exists but `grading.json` sits directly inside it — mirroring what the viewer already accepts. As a cheap defense-in-depth addition, `aggregate_benchmark.py` should also print a visible error when zero `grading.json` files are found at all, rather than silently returning all-zero stats.

I've verified a minimal patch along these lines works correctly across all three documented layouts (flat single-run, `run-N/`, and the legacy `runs/` wrapper) with no regressions, and I'm happy to open a PR with it if that's useful.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.