huggingface / huggingface/evaluate
Reduce memory footprint of `compute`
Open
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
Currently, when `compute` is called all data is loaded into memory and passed as a list. This can pose a bottleneck especially for data intensive modalities (e.g. images) or measurements of large datasets. As an alternative we could pass a generator (or something similar) object that iterates over the datasets as many metrics are calculated in for loops or list comprehensions. If necessary we can still convert such an object to a list and load it into memory.
What do you think? @lhoestq @sashavor
Contributor guide
Assessment
This issue has not been assessed yet.