huggingface / huggingface/lighteval
[FT] supporting train-time vs test-time metrics
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Issue encountered
I have reward functions where:
- At train time: stricter behavior, aka it's harder to get "correct" (1.0 reward)
- At test time: relaxed reward behavior
This is analogous to:
- Train time: high school students only being allowed non-graphing calculators during tests
- Test time: workforce lets employees use computers at will
From https://huggingface.co/docs/lighteval/en/quicktour#basic-usage:
> The syntax for the task specification might be a bit hard to grasp at first. The format is as follows:
> ```none
> {suite}|{task}|{num_few_shot}|{0 for strict `num_few_shots`, or 1 to allow a truncation if context size is too small}
> ```
This syntax well-supports few-shot behaviors, but does not support train-time vs test-time differences.
## Solution/Feature
I have a few ideas:
- Generalizing `num_few_shot` (third entry) to be arbitrary metadata
- Making a new CLI arg such as `--loose` or `--strict`
## Possible alternatives
Duplicating metrics by name, so one goes from `reward_function_x` to `reward_function_x_train` + `reward_function_x_test`. This feels hacky though, it's not using the framework to solve problems for me.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.