anthropics / anthropics/skills

run_eval.py: claude -p never triggers skills/commands (0% trigger rate across all queries)

未关闭
#556 12 条评论 7 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
176k
派生
20.8k
平均合并
7 小时 21 分钟
30 天内合并 PR
5

描述

## Bug Description

`run_eval.py` creates a command file in `.claude/commands/` with the skill description in YAML frontmatter, then runs `claude -p` with test queries to check if Claude invokes the `Skill` or `Read` tool for that command. In practice, **no query ever triggers the skill** — all should-trigger queries get 0/3 trigger rate, including an explicit "run /lucas:memory-layer please".

## Reproduction

Tested with 8 different skills (144 total queries across all eval sets). Every should-trigger query scored 0/3 triggers. Every should-not-trigger query also scored 0/3 (correctly, but vacuously).

Diagnostic test with maximally obvious trigger:
```json
[{"query": "I need to use the lucas memory layer skill right now, run /lucas:memory-layer please", "should_trigger": true}]
```
Result: 0/3 triggers.

## Environment
- Claude Code v2.1.71
- macOS Darwin 25.3.0
- Model: claude-opus-4-6
- Ran from external terminal (no active Claude Code session)

## Analysis

The issue appears to be that `claude -p` does not auto-trigger commands placed in `.claude/commands/` the same way an interactive Claude Code session triggers skills from `~/.claude/skills/`. Possible causes:

1. **Commands vs Skills**: Commands in `.claude/commands/` may not appear in Claude's `available_skills` list the same way skills do, or may only be invocable via explicit `/command-name` syntax
2. **`claude -p` mode**: The `-p` (pipe/prompt) mode may not load commands/skills at all, or may not present them to the model
3. **Simple query threshold**: Claude may not consult skills for queries it can handle directly — but even explicit "run this skill" failed, ruling this out as the sole cause

## Expected Behavior

At minimum, an explicit request like "run /skill-name please" should trigger the command (rate > 0/3). Description-matched queries should trigger at a reasonable rate.

## Suggested Fix

- Verify that `claude -p` actually loads and presents `.claude/commands/` files to the model
- Consider placing test skills in `~/.claude/skills/` instead of `.claude/commands/` if skills and commands are triggered differently
- Add a `--debug` flag to `run_eval.py` that dumps what Claude actually sees (system prompt, available skills list) to verify the command file is being presented

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。