huggingface / huggingface/lighteval

[FT] supporting train-time vs test-time metrics

Open
#798 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Issue encountered

I have reward functions where:

- At train time: stricter behavior, aka it's harder to get "correct" (1.0 reward)
- At test time: relaxed reward behavior

This is analogous to:

- Train time: high school students only being allowed non-graphing calculators during tests
- Test time: workforce lets employees use computers at will

From https://huggingface.co/docs/lighteval/en/quicktour#basic-usage:

> The syntax for the task specification might be a bit hard to grasp at first. The format is as follows:
> ```none
> {suite}|{task}|{num_few_shot}|{0 for strict `num_few_shots`, or 1 to allow a truncation if context size is too small}
> ```

This syntax well-supports few-shot behaviors, but does not support train-time vs test-time differences.

## Solution/Feature

I have a few ideas:

- Generalizing `num_few_shot` (third entry) to be arbitrary metadata
- Making a new CLI arg such as `--loose` or `--strict`

## Possible alternatives

Duplicating metrics by name, so one goes from `reward_function_x` to `reward_function_x_train` + `reward_function_x_test`. This feels hacky though, it's not using the framework to solve problems for me.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.