huggingface / huggingface/lighteval
[FT] Adding caching for each dataset run
Open
feature
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Issue encountered
When running large evals with many dataset configurations it is very painful to rerun everything in case something fails.
## Solution/Feature
It would be great if intermediate results could be cached, for example the computed metrics of each dataset.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.