anthropics / anthropics/claude-agent-sdk-python

to_openeval()/from_openeval(): portable eval dataset conversion for ResultMessage / AssistantMessage

Aberta
#1,231 0 comentários 0 reações 0 responsáveis Ver no GitHub
enhancement
Linguagem predominante
Python
Estrelas
8.1k
Forks
1.3k
Métricas de merge de PRs
Métricas de PR pendentes

Descrição

Hi maintainers,

I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open, provider-neutral JSON Schema spec for portable LLM eval test suites and result sets, so eval data isn't locked to one framework's format. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

For context on why I'm raising this: openai/openai-python#3619 (https://github.com/openai/openai-python/pull/3619 — open, not yet merged) has an OpenAI contributor building native `to_openeval()` / `from_openeval()` helpers directly onto that SDK's dataset/response types. I think the same pattern would fit claude-agent-sdk-python well, given how much of what an eval result needs is already sitting on your message types.

From `src/claude_agent_sdk/types.py`: `AssistantMessage.content` is a `list[ContentBlock]` (`TextBlock`, `ToolUseBlock`, `ToolResultBlock`, ...), and `ResultMessage` already carries `result`, `structured_output`, `total_cost_usd`, `usage`, `num_turns`, and `session_id`. A conversion helper could look roughly like:

```python
# sketch, not a PR
def to_openeval_result(
messages: list[AssistantMessage | UserMessage],
result: ResultMessage,
*,
test_case_id: str,
) -> dict:
final_text = "".join(
block.text
for msg in messages if isinstance(msg, AssistantMessage)
for block in msg.content if isinstance(block, TextBlock)
)
return {
"test_case_id": test_case_id,
"actual_output": final_text or (result.result or ""),
"metadata": {
"num_turns": result.num_turns,
"total_cost_usd": result.total_cost_usd,
"session_id": result.session_id,
},
}
```

which lines up with an EvalPort ResultSet entry: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/resultset.json (test_case_id, actual_output, grader_results, passed).

No pressure if this isn't a priority right now — just wanted it in front of you given the precedent forming in openai-python. Happy to sketch a fuller PR if there's interest.

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.