anthropics / anthropics/claude-agent-sdk-python
to_openeval()/from_openeval(): portable eval dataset conversion for ResultMessage / AssistantMessage
- Linguagem predominante
- Python
- Estrelas
- 8.1k
- Forks
- 1.3k
- Métricas de merge de PRs
- Métricas de PR pendentes
Descrição
Hi maintainers,
I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open, provider-neutral JSON Schema spec for portable LLM eval test suites and result sets, so eval data isn't locked to one framework's format. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
For context on why I'm raising this: openai/openai-python#3619 (https://github.com/openai/openai-python/pull/3619 — open, not yet merged) has an OpenAI contributor building native `to_openeval()` / `from_openeval()` helpers directly onto that SDK's dataset/response types. I think the same pattern would fit claude-agent-sdk-python well, given how much of what an eval result needs is already sitting on your message types.
From `src/claude_agent_sdk/types.py`: `AssistantMessage.content` is a `list[ContentBlock]` (`TextBlock`, `ToolUseBlock`, `ToolResultBlock`, ...), and `ResultMessage` already carries `result`, `structured_output`, `total_cost_usd`, `usage`, `num_turns`, and `session_id`. A conversion helper could look roughly like:
```python
# sketch, not a PR
def to_openeval_result(
messages: list[AssistantMessage | UserMessage],
result: ResultMessage,
*,
test_case_id: str,
) -> dict:
final_text = "".join(
block.text
for msg in messages if isinstance(msg, AssistantMessage)
for block in msg.content if isinstance(block, TextBlock)
)
return {
"test_case_id": test_case_id,
"actual_output": final_text or (result.result or ""),
"metadata": {
"num_turns": result.num_turns,
"total_cost_usd": result.total_cost_usd,
"session_id": result.session_id,
},
}
```
which lines up with an EvalPort ResultSet entry: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/resultset.json (test_case_id, actual_output, grader_results, passed).
No pressure if this isn't a priority right now — just wanted it in front of you given the precedent forming in openai-python. Happy to sketch a fuller PR if there's interest.
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Avaliação
Esta issue ainda não foi avaliada.