huggingface / huggingface/evaluate

Alternative backends for `evaluate`

Open
#356 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Currently `add`, `add_batch` as well as `compute` store data in Arrow format and then load it back to memory to feed it to `_compute`. There are a number of advantages to the current format:

- **typing**: the data's types are tested before written to the arrow table
- **memory**: especially when using `add` and `add_batch` no data is kept in memory and with minor adjustments even for computing the scores one could profit from Arrow's memory mapping (see #168)

However, the Arrow backend can also come with an overhead and heavily depends on `datasets`. For most applications this is not a big concern (e.g. tabular or text data) but can cause an slowdown e.g. in computer vision. A lightweight alternative could be to simply keep data in memory and pass it directly to `_compute`.

This is how it could work: during loading pass an additional kwarg that's `backend="arrow"` by default but can be changed to `backend="memory"` which then uses native Python. In the backend `add` and `add_batch` simply add data to a list and passes it to `_compute`. We could use this model also to phase out `datasets` by at some point replacing the defaults of `backend`.

**Other frameworks:** Using other frameworks (e.g. PyTorch or TF) for the metric computation will probably require extra code. We could define `_compute_tf` and `_compute_pt` methods in the metric modules. In `compute` we can check if those are defined for a certain module (these methods might not be implemented from the start or not feasible in these frameworks) and fallback to `_compute` if not.

I think we could do the `memory`/Python backend pretty quickly and then do the other frameworks in a second step.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.