anthropics / anthropics/claude-agent-sdk-python

to_openeval()/from_openeval(): portable eval dataset conversion for ResultMessage / AssistantMessage

未關閉
#1,231 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
Python
星號
8.1k
分支
1.3k
PR 合併指標
PR 指標待擷取

描述

Hi maintainers,

I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open, provider-neutral JSON Schema spec for portable LLM eval test suites and result sets, so eval data isn't locked to one framework's format. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

For context on why I'm raising this: openai/openai-python#3619 (https://github.com/openai/openai-python/pull/3619 — open, not yet merged) has an OpenAI contributor building native `to_openeval()` / `from_openeval()` helpers directly onto that SDK's dataset/response types. I think the same pattern would fit claude-agent-sdk-python well, given how much of what an eval result needs is already sitting on your message types.

From `src/claude_agent_sdk/types.py`: `AssistantMessage.content` is a `list[ContentBlock]` (`TextBlock`, `ToolUseBlock`, `ToolResultBlock`, ...), and `ResultMessage` already carries `result`, `structured_output`, `total_cost_usd`, `usage`, `num_turns`, and `session_id`. A conversion helper could look roughly like:

```python
# sketch, not a PR
def to_openeval_result(
messages: list[AssistantMessage | UserMessage],
result: ResultMessage,
*,
test_case_id: str,
) -> dict:
final_text = "".join(
block.text
for msg in messages if isinstance(msg, AssistantMessage)
for block in msg.content if isinstance(block, TextBlock)
)
return {
"test_case_id": test_case_id,
"actual_output": final_text or (result.result or ""),
"metadata": {
"num_turns": result.num_turns,
"total_cost_usd": result.total_cost_usd,
"session_id": result.session_id,
},
}
```

which lines up with an EvalPort ResultSet entry: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/resultset.json (test_case_id, actual_output, grader_results, passed).

No pressure if this isn't a priority right now — just wanted it in front of you given the precedent forming in openai-python. Happy to sketch a fuller PR if there's interest.

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。