huggingface / huggingface/lighteval

[FT] `MetricUseCase.FORMATTING` for reasoning models

Open
#803 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Issue encountered

With reasoning models such as DeepSeek R1, these models have a format reward applied to reasoning traces. From https://arxiv.org/pdf/2501.12948:

> **Format rewards**: In addition to the accuracy reward model, we employ a format reward model that enforces the model to put its thinking process between between ‘< think >’ and ‘’ tags.

(note I put space around `<>` so GitHub Flavored Markdown doesn't truncate the XML).

## Solution/Feature

A new category for `MetricUseCase` called `FORMATTING`, for these types of evaluations.

## Possible alternatives

There is an argument this format reward should be treated as an accuracy (`MetricUseCase.ACCURACY`), because format reward (when given `1.0` for matching, `0.0` for not matching), can be interpreted as a percentage of correctly formatted responses.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.