anthropics / anthropics/claude-plugins-official
skill-creator eval workflow docs disagree with benchmark/viewer schemas
- Dominant language
- Python
- Stars
- 36.3k
- Forks
- 4.1k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 539
Description
## Summary
The `skill-creator` workflow documentation describes eval artifacts in a layout and field naming that do not match the benchmark/viewer tooling.
This can cause a user or agent following `SKILL.md` to generate eval outputs that the benchmark script silently ignores.
## Problems
1. `SKILL.md` asks runs to save outputs under:
```text
/iteration-/eval-/with_skill/outputs/
```
But `scripts/aggregate_benchmark.py` only reads grading files under:
```text
/eval-//run-*/grading.json
```
Following the documented flat layout can produce a `benchmark.json` with no runs and exit code 0.
2. `SKILL.md` refers to an `assertions` field in `eval_metadata.json`, but `references/schemas.md`, `agents/grader.md`, and `scripts/aggregate_benchmark.py` use `expectations`.
3. Once the documented layout is aligned to include `run-*`, `eval-viewer/generate_review.py` also needs to find `eval_metadata.json` from the eval directory, not only from the run/config directory.
## Reproduction
Create a benchmark workspace following the documented flat layout:
```text
iteration-1/
eval-0/
with_skill/
grading.json
without_skill/
grading.json
```
Run:
```bash
python -m scripts.aggregate_benchmark iteration-1 --skill-name fixture
```
`benchmark.json` is generated, but contains no eval results:
```json
{
"metadata": {
"evals_run": []
},
"runs": []
}
```
Move the same `grading.json` files under `run-1/`:
```text
iteration-1/
eval-0/
with_skill/
run-1/
grading.json
without_skill/
run-1/
grading.json
```
Run the same command again. The benchmark now contains runs as expected.
## Expected behavior
The documentation and tooling should agree on one canonical eval artifact layout and JSON field naming.
## Proposed fix
- Document `eval-//run-*` as the eval artifact layout.
- Replace `assertions` with `expectations` in `SKILL.md`.
- Update `eval-viewer/generate_review.py` to search the run/config/eval ancestry for `eval_metadata.json`.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read SKILL.md, references/schemas.md, scripts/aggregate_benchmark.py, and eval-viewer/generate_review.py, then reproduce the documented flat-layout command with the supplied fixture structure. Done means the docs and tooling use the same run-* artifact layout and expectations field, and review generation finds eval_metadata.json from the eval ancestry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100