anthropics / anthropics/claude-cookbooks
Cookbook idea: a portable-eval-format example alongside evals/ and tool_evaluation/
- Linguagem predominante
- Jupyter Notebook
- Estrelas
- 52.7k
- Forks
- 6.3k
- Merge médio
- 25min
- PRs com merge (30d)
- 6
Descrição
Hi — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open, versioned JSON spec for LLM evaluation datasets (test suites, test cases, graders, result sets), with JSON Schemas and Python/TypeScript SDKs that validate against them. Posting this as an issue per CONTRIBUTING.md rather than opening a PR unannounced.
What caught my eye browsing this repo: `tool_evaluation/evaluation.xml` is a small hand-rolled format — a flat list of `......` pairs consumed by `tool_evaluation.ipynb`. And `evals/agentic_search/reproduce_agentic_search_benchmarks.ipynb` reproduces benchmark results with its own loading/scoring plumbing. Both are exactly the kind of "every notebook invents its own eval-set shape" situation EvalPort exists to avoid — a suite is just JSON with `id`, `input`, `expected_output`, and a `graders` array, so it round-trips through other tools instead of being notebook-specific.
Concretely, the `evaluation.xml` cases would map almost 1:1 onto EvalPort test cases:
```json
{
"version": "1.0.0",
"id": "tool-evaluation-math",
"test_cases": [
{
"id": "compound-interest-1",
"input": "Calculate the compound interest on $10,000 invested at 5% annual interest rate, compounded monthly for 3 years...",
"expected_output": "11614.72",
"graders": ["exact-match"]
}
]
}
```
I'd be glad to send a small PR — either a new notebook under `misc/` or `evals/` showing "load an EvalPort suite → run it against Claude → write an EvalPort result set," or just a converted version of the existing `evaluation.xml` — whichever fits your roadmap better. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
No pressure at all if this isn't a direction you want cookbook content to go — happy to close this out either way. Thanks for maintaining such a useful set of notebooks.
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.