anthropics / anthropics/claude-agent-sdk-python

to_openeval()/from_openeval(): portable eval dataset conversion for ResultMessage / AssistantMessage

Offen
#1,231 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
enhancement
Vorherrschende Sprache
Python
Sterne
8.1k
Forks
1.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

Hi maintainers,

I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open, provider-neutral JSON Schema spec for portable LLM eval test suites and result sets, so eval data isn't locked to one framework's format. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

For context on why I'm raising this: openai/openai-python#3619 (https://github.com/openai/openai-python/pull/3619 — open, not yet merged) has an OpenAI contributor building native `to_openeval()` / `from_openeval()` helpers directly onto that SDK's dataset/response types. I think the same pattern would fit claude-agent-sdk-python well, given how much of what an eval result needs is already sitting on your message types.

From `src/claude_agent_sdk/types.py`: `AssistantMessage.content` is a `list[ContentBlock]` (`TextBlock`, `ToolUseBlock`, `ToolResultBlock`, ...), and `ResultMessage` already carries `result`, `structured_output`, `total_cost_usd`, `usage`, `num_turns`, and `session_id`. A conversion helper could look roughly like:

```python
# sketch, not a PR
def to_openeval_result(
messages: list[AssistantMessage | UserMessage],
result: ResultMessage,
*,
test_case_id: str,
) -> dict:
final_text = "".join(
block.text
for msg in messages if isinstance(msg, AssistantMessage)
for block in msg.content if isinstance(block, TextBlock)
)
return {
"test_case_id": test_case_id,
"actual_output": final_text or (result.result or ""),
"metadata": {
"num_turns": result.num_turns,
"total_cost_usd": result.total_cost_usd,
"session_id": result.session_id,
},
}
```

which lines up with an EvalPort ResultSet entry: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/resultset.json (test_case_id, actual_output, grader_results, passed).

No pressure if this isn't a priority right now — just wanted it in front of you given the precedent forming in openai-python. Happy to sketch a fuller PR if there's interest.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.