anthropics / anthropics/claude-code-action

skill-creator eval scripts (run_eval.py) cannot run inside claude-code-action — requires nested claude CLI

Open
#1,073 0 comments 0 reactions 0 assignees View on GitHub
feature-request p3
Dominant language
TypeScript
Stars
8.9k
Forks
2.1k
Avg merge
3d 9h
Merged PRs (30d)
10

Description

## Problem

The `skill-creator` plugin from `claude-plugins-official` includes Python eval scripts (`run_eval.py`, `quick_validate.py`) that are designed to measure skill trigger accuracy. However, these scripts **cannot run inside `claude-code-action`** because they invoke `claude -p` as a subprocess.

Inside the action, Claude is already running as the agent via the Agent SDK. There's no standalone `claude` CLI binary available on the runner for `run_eval.py` to shell out to.

This means there's **no way to run skill-creator evals in CI** using the official tooling, despite the plugin being designed for exactly this use case.

## What happens

1. Configure a workflow with `plugins: skill-creator`
2. Prompt Claude to run the eval pipeline using `run_eval.py`
3. Claude attempts to run the script, but `claude -p` is not found / not available
4. The workflow completes with no eval output

## Reproduction

```yaml
- uses: anthropics/claude-code-action@v1
with:
plugins: skill-creator
prompt: |
Run the eval pipeline for skills using run_eval.py
```

## Expected behavior

Either:
1. **The `claude` CLI should be available on the runner** when using `claude-code-action`, so plugin scripts that depend on it can work
2. **`run_eval.py` should have an alternative mode** that uses the Anthropic API directly instead of shelling out to `claude -p`
3. **Document this limitation** so users don't build CI pipelines expecting eval scripts to work in the action

## Workaround

We rewrote our eval workflow prompt to have Claude assess trigger accuracy inline (reading the skill description and test cases, then reasoning about whether each query would trigger) rather than using the Python scripts. This works but loses the rigor of actual Claude-in-the-loop evaluation.

## Context

- `anthropics/claude-plugins-official` — `plugins/skill-creator/skills/skill-creator/scripts/run_eval.py`
- `run_eval.py` calls `subprocess.Popen(["claude", "-p", ...])` at line ~85
- Related: #1071 (PR-triggered workflows produce no visible output)

Contributor guide

Open the contributing guide

Research direction

Start by reading plugins/skill-creator/skills/skill-creator/scripts/run_eval.py, especially the subprocess call near line 85, then inspect how claude-code-action sets up its runner and Agent SDK. Reproduce the missing CLI behavior with the workflow shown in the issue. Done means choosing and implementing one supported path—CLI availability, an API-based mode, or clear limitation documentation—and verifying the eval workflow's result.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python, typescript
Domain
ci-cd, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.