anthropics / anthropics/claude-plugins-official

skill-creator eval workflow docs disagree with benchmark/viewer schemas

Open
#3,926 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
36.3k
Forks
4.1k
Avg merge
2d 14h
Merged PRs (30d)
539

Description

## Summary

The `skill-creator` workflow documentation describes eval artifacts in a layout and field naming that do not match the benchmark/viewer tooling.

This can cause a user or agent following `SKILL.md` to generate eval outputs that the benchmark script silently ignores.

## Problems

1. `SKILL.md` asks runs to save outputs under:

```text
/iteration-/eval-/with_skill/outputs/
```

But `scripts/aggregate_benchmark.py` only reads grading files under:

```text
/eval-//run-*/grading.json
```

Following the documented flat layout can produce a `benchmark.json` with no runs and exit code 0.

2. `SKILL.md` refers to an `assertions` field in `eval_metadata.json`, but `references/schemas.md`, `agents/grader.md`, and `scripts/aggregate_benchmark.py` use `expectations`.

3. Once the documented layout is aligned to include `run-*`, `eval-viewer/generate_review.py` also needs to find `eval_metadata.json` from the eval directory, not only from the run/config directory.

## Reproduction

Create a benchmark workspace following the documented flat layout:

```text
iteration-1/
eval-0/
with_skill/
grading.json
without_skill/
grading.json
```

Run:

```bash
python -m scripts.aggregate_benchmark iteration-1 --skill-name fixture
```

`benchmark.json` is generated, but contains no eval results:

```json
{
"metadata": {
"evals_run": []
},
"runs": []
}
```

Move the same `grading.json` files under `run-1/`:

```text
iteration-1/
eval-0/
with_skill/
run-1/
grading.json
without_skill/
run-1/
grading.json
```

Run the same command again. The benchmark now contains runs as expected.

## Expected behavior

The documentation and tooling should agree on one canonical eval artifact layout and JSON field naming.

## Proposed fix

- Document `eval-//run-*` as the eval artifact layout.
- Replace `assertions` with `expectations` in `SKILL.md`.
- Update `eval-viewer/generate_review.py` to search the run/config/eval ancestry for `eval_metadata.json`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read SKILL.md, references/schemas.md, scripts/aggregate_benchmark.py, and eval-viewer/generate_review.py, then reproduce the documented flat-layout command with the supplied fixture structure. Done means the docs and tooling use the same run-* artifact layout and expectations field, and review generation finds eval_metadata.json from the eval ancestry.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.