huggingface / huggingface/lighteval
[FT] `MetricUseCase.FORMATTING` for reasoning models
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Issue encountered
With reasoning models such as DeepSeek R1, these models have a format reward applied to reasoning traces. From https://arxiv.org/pdf/2501.12948:
> **Format rewards**: In addition to the accuracy reward model, we employ a format reward model that enforces the model to put its thinking process between between ‘< think >’ and ‘’ tags.
(note I put space around `<>` so GitHub Flavored Markdown doesn't truncate the XML).
## Solution/Feature
A new category for `MetricUseCase` called `FORMATTING`, for these types of evaluations.
## Possible alternatives
There is an argument this format reward should be treated as an accuracy (`MetricUseCase.ACCURACY`), because format reward (when given `1.0` for matching, `0.0` for not matching), can be interpreted as a percentage of correctly formatted responses.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.