huggingface / huggingface/evaluate

Reduce memory footprint of `compute`

Open
#168 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Currently, when `compute` is called all data is loaded into memory and passed as a list. This can pose a bottleneck especially for data intensive modalities (e.g. images) or measurements of large datasets. As an alternative we could pass a generator (or something similar) object that iterates over the datasets as many metrics are calculated in for loops or list comprehensions. If necessary we can still convert such an object to a list and load it into memory.

What do you think? @lhoestq @sashavor

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.