anthropics / anthropics/claude-agent-sdk-python

to_openeval()/from_openeval(): portable eval dataset conversion for ResultMessage / AssistantMessage

Abierto
#1,231 0 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement
Lenguaje dominante
Python
Estrellas
8.1k
Forks
1.3k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

Hi maintainers,

I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open, provider-neutral JSON Schema spec for portable LLM eval test suites and result sets, so eval data isn't locked to one framework's format. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

For context on why I'm raising this: openai/openai-python#3619 (https://github.com/openai/openai-python/pull/3619 — open, not yet merged) has an OpenAI contributor building native `to_openeval()` / `from_openeval()` helpers directly onto that SDK's dataset/response types. I think the same pattern would fit claude-agent-sdk-python well, given how much of what an eval result needs is already sitting on your message types.

From `src/claude_agent_sdk/types.py`: `AssistantMessage.content` is a `list[ContentBlock]` (`TextBlock`, `ToolUseBlock`, `ToolResultBlock`, ...), and `ResultMessage` already carries `result`, `structured_output`, `total_cost_usd`, `usage`, `num_turns`, and `session_id`. A conversion helper could look roughly like:

```python
# sketch, not a PR
def to_openeval_result(
messages: list[AssistantMessage | UserMessage],
result: ResultMessage,
*,
test_case_id: str,
) -> dict:
final_text = "".join(
block.text
for msg in messages if isinstance(msg, AssistantMessage)
for block in msg.content if isinstance(block, TextBlock)
)
return {
"test_case_id": test_case_id,
"actual_output": final_text or (result.result or ""),
"metadata": {
"num_turns": result.num_turns,
"total_cost_usd": result.total_cost_usd,
"session_id": result.session_id,
},
}
```

which lines up with an EvalPort ResultSet entry: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/resultset.json (test_case_id, actual_output, grader_results, passed).

No pressure if this isn't a priority right now — just wanted it in front of you given the precedent forming in openai-python. Happy to sketch a fuller PR if there's interest.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.