anthropics / anthropics/skills
skill-creator: eval loop rough edges — grading paths, Bash blocking, aggregate_benchmark.py bugs
- Dominant language
- Python
- Stars
- 176k
- Forks
- 20.8k
- Avg merge
- 7h 21m
- Merged PRs (30d)
- 5
Description
## Context
We ran the skill-creator eval loop on a custom `az-cli` skill over 2 iterations (3 eval cases each, hitting live Azure AD) and encountered several bugs and usability issues in the skill-creator itself. These affect any user running the eval/benchmark workflow.
Checked existing issues — #490 (workspace location) and #470 (security test failure) are related but don't overlap with the items below.
## 1. Grader subagents write `grading.json` to wrong paths
Graders sometimes save `grading.json` to the eval directory root (e.g., `group-membership-check/grading.json`) instead of the config subdirectory (e.g., `group-membership-check/with_skill/grading.json`). The viewer and `aggregate_benchmark.py` both expect the file inside the config subdirectory (`with_skill/` or `without_skill/`).
The SKILL.md instructions say to save to "each run directory" but this is ambiguous — graders interpret it as the eval case directory rather than the config-specific run directory.
**Impact**: Benchmark viewer shows missing results; aggregate script skips the run entirely.
## 2. Baseline (`without_skill`) runs sometimes have Bash blocked
In our first iteration, one baseline subagent had the Bash tool entirely blocked and couldn't execute any CLI commands. It was graded on "documented intent" only, which produced a misleading 100% pass rate for the baseline.
We worked around this by adding explicit "You MUST use the Bash tool to execute commands" instructions to subagent prompts, but the skill-creator should ensure Bash is available by default for both `with_skill` and `without_skill` runs.
**Impact**: Baseline scores are inflated, making skill improvement appear smaller than it actually is (or negative).
## 3. `aggregate_benchmark.py` only finds `eval-*` directories
The script globs for `eval-*` prefixed directories:
```python
eval_dirs = sorted(iteration_path.glob("eval-*"))
```
However, the SKILL.md explicitly encourages descriptive names:
> Give each eval a descriptive name based on what it's testing — not just eval-0
These two instructions contradict each other. Any eval with a descriptive name (e.g., `group-membership-check`, `role-assignment-list`) is silently skipped by the aggregate script.
**Fix**: The script should iterate over all subdirectories that contain a `config.json`, rather than relying on the `eval-*` prefix.
## 4. `aggregate_benchmark.py` crashes on non-directory entries
The script calls `.iterdir()` on everything in the iteration directory without filtering to directories:
```python
for eval_dir in eval_dirs:
# assumes eval_dir is a directory
for config_dir in eval_dir.iterdir():
...
```
A screenshot, `.DS_Store`, or any other file in the directory causes a `NotADirectoryError` crash.
**Fix**: Add `if entry.is_dir()` checks before iterating.
## 5. `grading.json` field names not enforced
The SKILL.md specifies these fields for grading criteria:
- `text` — criterion description
- `passed` — boolean
- `evidence` — supporting detail
But grader subagents sometimes use variant field names (`name`/`met`/`details`). The viewer silently shows empty cells when the expected field names don't match, with no warning.
**Impact**: Results look like they were never graded. User has to manually inspect the JSON to discover the mismatch.
**Suggestion**: Either validate field names in the viewer/aggregate script (with a clear error), or make the grader prompt more explicit about the exact schema required.
---
## Environment
- Claude Code with skill-creator skill
- 2 iterations, 3 eval cases each
- Skill under test: custom `az-cli` skill calling live Azure AD via `az` CLI
- macOS, zsh
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.