huggingface / huggingface/lighteval
[BUG] Cache management failed for tasks using sample_params to update function attrs
Open
bug
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
Metric:__call__ used "&".join(f"{k}={v}" for k, v in sample_params.items()) to update metric name, but the value might be function, which will change for each run.
## To Reproduce
Task that use sample_params, such as:
leaderboard:gsm8k:0
metrics=[
Metrics.exact_match(sample_params={"normalize_gold": gsm8k_normalizer, "normalize_pred": gsm8k_normalizer})
],
## Expected behavior
Cache should work
## Version info
main branch
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.