anthropics / anthropics/claude-code-action
skill-creator eval scripts (run_eval.py) cannot run inside claude-code-action — requires nested claude CLI
- Dominant language
- TypeScript
- Stars
- 8.9k
- Forks
- 2.1k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 10
Description
## Problem
The `skill-creator` plugin from `claude-plugins-official` includes Python eval scripts (`run_eval.py`, `quick_validate.py`) that are designed to measure skill trigger accuracy. However, these scripts **cannot run inside `claude-code-action`** because they invoke `claude -p` as a subprocess.
Inside the action, Claude is already running as the agent via the Agent SDK. There's no standalone `claude` CLI binary available on the runner for `run_eval.py` to shell out to.
This means there's **no way to run skill-creator evals in CI** using the official tooling, despite the plugin being designed for exactly this use case.
## What happens
1. Configure a workflow with `plugins: skill-creator`
2. Prompt Claude to run the eval pipeline using `run_eval.py`
3. Claude attempts to run the script, but `claude -p` is not found / not available
4. The workflow completes with no eval output
## Reproduction
```yaml
- uses: anthropics/claude-code-action@v1
with:
plugins: skill-creator
prompt: |
Run the eval pipeline for skills using run_eval.py
```
## Expected behavior
Either:
1. **The `claude` CLI should be available on the runner** when using `claude-code-action`, so plugin scripts that depend on it can work
2. **`run_eval.py` should have an alternative mode** that uses the Anthropic API directly instead of shelling out to `claude -p`
3. **Document this limitation** so users don't build CI pipelines expecting eval scripts to work in the action
## Workaround
We rewrote our eval workflow prompt to have Claude assess trigger accuracy inline (reading the skill description and test cases, then reasoning about whether each query would trigger) rather than using the Python scripts. This works but loses the rigor of actual Claude-in-the-loop evaluation.
## Context
- `anthropics/claude-plugins-official` — `plugins/skill-creator/skills/skill-creator/scripts/run_eval.py`
- `run_eval.py` calls `subprocess.Popen(["claude", "-p", ...])` at line ~85
- Related: #1071 (PR-triggered workflows produce no visible output)
Contributor guide
Research direction
Start by reading plugins/skill-creator/skills/skill-creator/scripts/run_eval.py, especially the subprocess call near line 85, then inspect how claude-code-action sets up its runner and Agent SDK. Reproduce the missing CLI behavior with the workflow shown in the issue. Done means choosing and implementing one supported path—CLI availability, an API-based mode, or clear limitation documentation—and verifying the eval workflow's result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python, typescript
- Domain
- ci-cd, devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100