anthropics / anthropics/claude-code

plugin eval: scope `target: trace` to the graded run's own events

Đang mở
#92,017 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
area:plugins enhancement
Ngôn ngữ chính
Python
Star
145k
Fork
23.1k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

**Is your feature request related to a problem?**

A `target: trace` regex grader reads the whole trace file. A trace can hold more than one
terminating `result` record: across 125 trace files from our runs we found 23 with more than one,
including one holding subtypes `success`, `error_max_turns`, `success` — all three under a single
session id, in a trace whose own run reported `turns: 6` and `error: null`. A grader asserting
`not_contains: error_(max_turns|during_execution|max_budget_usd)`, which is the only way to detect
an abort from inside a grader, then fails a run that finished normally, with a complete reply.

We have seen this once across roughly 125 runs; it always fails in the safe direction, but it puts
a spurious red on the gate.

**Describe the solution you'd like**

Either scope `target: trace` to the events of the run being graded (the last session, or the
events between the run's own start and its terminating `result`), or expose the run record's
`error` and result `subtype` as a grader target directly, so an abort detector reads one field
instead of grepping a file.

**Describe alternatives you've considered**

Reading the run record's `error` field in a post-run script (we do this for authentication
failures, which end with a `success` subtype and are invisible to any grader target). Telling
readers to check `error` before believing a detector red, which is a note for a human rather than
a fix.

**Additional context**

CLI 2.1.260, `plugin eval` under `CLAUDE_CODE_WALNUT_SPIRE`.

We do not know what causes a single trace file to accumulate several terminating `result` records
under one session id — that may itself be the underlying bug.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Hướng nghiên cứu

Start at the plugin eval `target: trace` grader and reproduce the issue with a trace file containing multiple terminating `result` records. Inspect how the graded run's events and run record are selected; done means the grader can reliably inspect only that run's events or directly target its `error` and result `subtype`.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
cli, testing-qa
Loại issue
Tính năng
Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
35/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.