anthropics / anthropics/claude-cookbooks

Cookbook idea: a portable-eval-format example alongside evals/ and tool_evaluation/

Aberta
#845 0 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Jupyter Notebook
Estrelas
52.7k
Forks
6.3k
Merge médio
25min
PRs com merge (30d)
6

Descrição

Hi — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open, versioned JSON spec for LLM evaluation datasets (test suites, test cases, graders, result sets), with JSON Schemas and Python/TypeScript SDKs that validate against them. Posting this as an issue per CONTRIBUTING.md rather than opening a PR unannounced.

What caught my eye browsing this repo: `tool_evaluation/evaluation.xml` is a small hand-rolled format — a flat list of `......` pairs consumed by `tool_evaluation.ipynb`. And `evals/agentic_search/reproduce_agentic_search_benchmarks.ipynb` reproduces benchmark results with its own loading/scoring plumbing. Both are exactly the kind of "every notebook invents its own eval-set shape" situation EvalPort exists to avoid — a suite is just JSON with `id`, `input`, `expected_output`, and a `graders` array, so it round-trips through other tools instead of being notebook-specific.

Concretely, the `evaluation.xml` cases would map almost 1:1 onto EvalPort test cases:

```json
{
"version": "1.0.0",
"id": "tool-evaluation-math",
"test_cases": [
{
"id": "compound-interest-1",
"input": "Calculate the compound interest on $10,000 invested at 5% annual interest rate, compounded monthly for 3 years...",
"expected_output": "11614.72",
"graders": ["exact-match"]
}
]
}
```

I'd be glad to send a small PR — either a new notebook under `misc/` or `evals/` showing "load an EvalPort suite → run it against Claude → write an EvalPort result set," or just a converted version of the existing `evaluation.xml` — whichever fits your roadmap better. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md

No pressure at all if this isn't a direction you want cookbook content to go — happy to close this out either way. Thanks for maintaining such a useful set of notebooks.

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.