huggingface / huggingface/evaluate
EvalPort export/import helpers for bridging evaluate's aggregate-only API to portable per-example results
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
### Motivation
I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open, framework-agnostic JSON format for LLM evaluation test cases, graders, and results (schema-validated `Suite`/`ResultSet` documents — think "a portable file format eval tools can read/write instead of everyone inventing their own"). It already has independently-tested adapter packages for LangSmith, MLflow, Ragas, Braintrust, Arize Phoenix, Weave, and 16 other frameworks.
While building the `evaluate` one, I ran into a real, specific friction point that isn't a bug in `evaluate` — it's a genuine shape mismatch worth flagging directly. `EvaluationModule.compute(predictions=..., references=...)` always returns a whole-batch aggregate (confirmed against 0.4.6: `exact_match`, `accuracy`, `f1`, `bleu` all behave this way), which is exactly right for `evaluate`'s own purpose. But anyone trying to export `evaluate` results into a per-example format (for storing in a dataset, diffing two model versions example-by-example, or feeding a downstream eval-tracking tool) has no supported path to get there — you either write your own per-example loop yourself, or you're stuck with only the aggregate.
I already solved this on my side (real per-example `.compute()` calls, one call per example, never interpolated from the aggregate — no fabricated numbers) and it works well enough that I'd like to check whether something like it belongs upstream, since I'd guess I'm not the only person who's hit this.
### Feature description
A small, optional helper — something like `evaluate.compute_per_example(metric, predictions, references, **kwargs)` — that returns both the real whole-batch aggregate `evaluate` already computes today *and* a list of real per-example scores, obtained by calling the same metric's `.compute()` once per individual `(prediction, reference)` pair rather than once for the whole batch. No new metric logic, no new dependency — just a thin convenience wrapper around the existing `.compute()` call pattern, with a clear docstring caveat that per-example scoring isn't a meaningful concept for corpus-level statistics (classic BLEU being the standard example) the way it is for `exact_match`/`accuracy`/per-item `f1`.
If this is something the maintainers would want as a PR, I'm happy to open one against the API shape you'd prefer (a module-level function, a method on `EvaluationModule`, or something else) — I already have a tested reference implementation ([`compute_per_example`](https://github.com/adhabnr-ux/evalport/blob/main/adapters/huggingface-evaluate-openeval-adapter/src/huggingface_evaluate_openeval_adapter/__init__.py)) I built and validated for the standalone EvalPort adapter that I'd adapt to fit `evaluate`'s own conventions rather than paste in as-is.
### Code snippet demonstrating current vs. desired use
```python
import evaluate
metric = evaluate.load("exact_match")
predictions = ["Paris", "5", "Berlin"]
references = ["Paris", "4", "Berlin"]
# Today: only the aggregate is available from the public API
metric.add_batch(predictions=predictions, references=references)
result = metric.compute()
# {"exact_match": 0.666...} -- which examples failed? not recoverable from this alone
# Desired: an opt-in helper that also returns real per-example scores
aggregate, per_example = evaluate.compute_per_example("exact_match", predictions, references)
# aggregate == {"exact_match": 0.666...} <- same real batch call as today
# per_example == [1.0, 0.0, 1.0] <- one real .compute() call per example, not interpolated
```
### Additional context
For anyone curious what this looks like end-to-end (including the honesty caveat about corpus-level metrics), the standalone EvalPort adapter is public and tested: [huggingface-evaluate-openeval-adapter](https://github.com/adhabnr-ux/evalport/tree/main/adapters/huggingface-evaluate-openeval-adapter) (25/25 tests passing against the real installed `evaluate` package, README has a full "why this looks different" writeup). Not asking for an EvalPort integration itself — just flagging a real API gap I hit and offering to help close it if it's wanted upstream. Totally fine if this isn't a direction the maintainers want to take `evaluate` in; the adapter works standalone either way.
Contributor guide
Research direction
Start by reading EvaluationModule.compute and the public evaluate.load and metric.add_batch usage described in the issue. Compare the tested reference implementation in EvalPort's adapters/huggingface-evaluate-openeval-adapter with evaluate's conventions, including its 25-test suite. Done means an agreed public API that preserves the batch aggregate, performs real per-example compute calls, documents corpus-level limitations, and has upstream tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100