huggingface / huggingface/lighteval

[Proposal] OpenEval Import/Export Support

Open
#1,318 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Overview

We have launched [OpenEval](https://github.com/adhabnr-ux/openeval) (v1.0.0-rc.1), an open standard (Apache 2.0) for portable LLM evaluation datasets. Every eval framework uses incompatible formats — OpenEval provides a standard JSON format + converters + SDKs (TypeScript + Python).

## What We're Proposing

Add `to_openeval()` and `from_openeval()` methods (~50-100 lines using our SDK).

## Resources

- **Spec:** https://github.com/adhabnr-ux/openeval/blob/main/spec/SPEC.md
- **npm SDK:** `npm install @openeval/sdk`
- **PyPI SDK:** `pip install openeval`

We would love to collaborate.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the OpenEval spec at spec/SPEC.md and reviewing the TypeScript and Python SDK resources linked in the issue. Then locate the appropriate Lighteval integration entry points; done means adding to_openeval() and from_openeval() methods that exchange the proposed standard JSON format.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, typescript
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.