allenai / allenai/olmes

Optional EvalPort export for OLMES task instances and predictions

Abierto
#58 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
395
Forks
105
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

Hi OLMES team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open, framework-agnostic JSON spec for portable LLM eval datasets and results (Apache 2.0, a `TestCase`/`Suite`/`ResultSet` schema with a real JSON Schema validator, so a dataset or a graded run can move between tools without hand-writing a converter each time).

I read `OUTPUT_FORMATS.md` directly rather than guessing at fields. The per-instance request document:

```json
{
"id": "Mercury_7175875",
"query": "Question: An astronomer observes that a planet rotates faster after a meteorite impact...\nAnswer:",
"choices": ["Planetary density will decrease.", "Planetary years will become longer.", ...],
"gold": 2
}
```

and the predictions file, per instance:

```json
{
"doc_id": 0, "native_id": "Mercury_7175875",
"metrics": {"predicted_index_uncond": 2, "acc_uncond": 1, ...},
"label": 2, "task_hash": "...", "model_hash": "..."
}
```

plus the task-level `metrics.json` with `primary_score` and full `task_config`/`model_config`.

This maps very directly onto EvalPort: `query` → `TestCase.input`, `choices` → the option set, `gold` → `expected_output`, `id`/`native_id` → `TestCase.id`, and `task_config` (`dataset_path`, `num_shots`, `primary_metric`, etc.) → `metadata`. On the results side, `task-*-predictions.jsonl` (per-instance `metrics`, `label`, model output) plus `metrics.json`'s `primary_score` map onto EvalPort's `ResultSet`, which already has a defined shape for per-case pass/fail plus aggregate metrics.

Two ways I could see this landing, no strong preference:

1. A standalone `olmes-openeval-adapter` package in the EvalPort repo, converting OLMES's `task-*-requests.jsonl`/`predictions.jsonl`/`metrics.json` outputs to/from EvalPort `Suite`/`ResultSet`. Zero footprint on this repo.
2. A small optional export utility inside `oe_eval/` if maintainers would rather it live here.

Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock. Let me know which direction you'd prefer, or if this isn't a fit for the roadmap right now — no worries either way.

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Start with OUTPUT_FORMATS.md and the existing oe_eval/ entry points to understand how task-*-requests.jsonl, task-*-predictions.jsonl, and metrics.json are produced. Compare the proposed standalone adapter and in-repository utility with EvalPort's actual JSON Schema; done means an agreed integration path and real conversion tests validating task instances, predictions, and aggregate metrics against that schema.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
data, machine-learning
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Activo
Claridad
Bastante claro
Aptitud para principiantes
42/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.