huggingface / huggingface/evaluate
Alternative backends for `evaluate`
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
Currently `add`, `add_batch` as well as `compute` store data in Arrow format and then load it back to memory to feed it to `_compute`. There are a number of advantages to the current format:
- **typing**: the data's types are tested before written to the arrow table
- **memory**: especially when using `add` and `add_batch` no data is kept in memory and with minor adjustments even for computing the scores one could profit from Arrow's memory mapping (see #168)
However, the Arrow backend can also come with an overhead and heavily depends on `datasets`. For most applications this is not a big concern (e.g. tabular or text data) but can cause an slowdown e.g. in computer vision. A lightweight alternative could be to simply keep data in memory and pass it directly to `_compute`.
This is how it could work: during loading pass an additional kwarg that's `backend="arrow"` by default but can be changed to `backend="memory"` which then uses native Python. In the backend `add` and `add_batch` simply add data to a list and passes it to `_compute`. We could use this model also to phase out `datasets` by at some point replacing the defaults of `backend`.
**Other frameworks:** Using other frameworks (e.g. PyTorch or TF) for the metric computation will probably require extra code. We could define `_compute_tf` and `_compute_pt` methods in the metric modules. In `compute` we can check if those are defined for a certain module (these methods might not be implemented from the start or not feasible in these frameworks) and fallback to `_compute` if not.
I think we could do the `memory`/Python backend pretty quickly and then do the other frameworks in a second step.
Contributor guide
Assessment
This issue has not been assessed yet.