huggingface / huggingface/evaluate

Cache results from `Evaluator`

Open
#126 3 comments 1 reaction 2 assignees Claimed by @mathemakitten View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

When running an evaluation with the `Evaluator` class, it would be great to cache the results (e.g. store them as an Arrow dataset) so that one doesn't have to wait to recompute everything each time.

This would also give `evaluate` a similar developer experience to `datasets` and allow users to leverage the intuitions from one lib to the other :)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.