anthropics / anthropics/claude-plugins-official

skill-creator: aggregate_benchmark.py silently zeroes total_tokens and hardcodes runs_per_configuration

Đang mở
#5,723 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
36.2k
Fork
4.1k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

`skills/skill-creator/scripts/aggregate_benchmark.py` has two metric-corrupting bugs, both reproducible by reading the code:

**1. `total_tokens` is only read when timing is missing from grading.json** (~line 140). The token count lives inside the fallback branch:

```python
result["time_seconds"] = timing.get("total_duration_seconds", 0.0)
timing_file = run_dir / "timing.json"
if result["time_seconds"] == 0.0 and timing_file.exists():
...
result["time_seconds"] = timing_data.get("total_duration_seconds", 0.0)
result["tokens"] = timing_data.get("total_tokens", 0)
```

When `grading.json` carries timing (the normal case), the branch never runs and `tokens` stays 0 — the benchmark reports zero token cost with no warning. Fix: read `total_tokens` unconditionally from `timing.json` (or from grading.json's timing block if it carries one).

**2. `runs_per_configuration` is hardcoded to `3`** (~line 270, in `benchmark["metadata"]`), regardless of how many runs actually aggregated. Any benchmark run with a different N reports wrong metadata, and downstream variance analysis that trusts it divides by the wrong count. Fix: derive it from the runs actually collected.

Observed on plugin cache version `ed404106fcd8`, running benchmarks locally on real skills; both persisted across a plugin update.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.