huggingface / huggingface/lighteval

[BUG] Cache management failed for tasks using sample_params to update function attrs

Open
#1,026 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
Metric:__call__ used "&".join(f"{k}={v}" for k, v in sample_params.items()) to update metric name, but the value might be function, which will change for each run.

## To Reproduce
Task that use sample_params, such as:
leaderboard:gsm8k:0

metrics=[
Metrics.exact_match(sample_params={"normalize_gold": gsm8k_normalizer, "normalize_pred": gsm8k_normalizer})
],

## Expected behavior
Cache should work

## Version info
main branch

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.