huggingface / huggingface/lighteval

[EVAL] Add Multi-IF

Open
#871 0 comments 0 reactions 1 assignee Claimed by @clefourrier View on GitHub
science-team
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Evaluation short description

Multi-IF has become one the multilingual analogue of the popular IFEval benchmark and is widely used in most recent model releases like Qwen3, SmolLM3 etc.

It requires LLMs to follow verifiable instructions across 3 turns of user prompts and thus has the added benefit of testing LLMs' ability to follow more than one turn.

## Evaluation metadata
Provide all available
- Paper url: https://arxiv.org/abs/2410.15553
- Github url: https://github.com/facebookresearch/Multi-IF
- Dataset url: https://huggingface.co/datasets/facebook/Multi-IF

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.