Optional EvalPort export for OLMES task instances and predictions
- Lenguaje dominante
- Python
- Estrellas
- 395
- Forks
- 105
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
Hi OLMES team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open, framework-agnostic JSON spec for portable LLM eval datasets and results (Apache 2.0, a `TestCase`/`Suite`/`ResultSet` schema with a real JSON Schema validator, so a dataset or a graded run can move between tools without hand-writing a converter each time).
I read `OUTPUT_FORMATS.md` directly rather than guessing at fields. The per-instance request document:
```json
{
"id": "Mercury_7175875",
"query": "Question: An astronomer observes that a planet rotates faster after a meteorite impact...\nAnswer:",
"choices": ["Planetary density will decrease.", "Planetary years will become longer.", ...],
"gold": 2
}
```
and the predictions file, per instance:
```json
{
"doc_id": 0, "native_id": "Mercury_7175875",
"metrics": {"predicted_index_uncond": 2, "acc_uncond": 1, ...},
"label": 2, "task_hash": "...", "model_hash": "..."
}
```
plus the task-level `metrics.json` with `primary_score` and full `task_config`/`model_config`.
This maps very directly onto EvalPort: `query` → `TestCase.input`, `choices` → the option set, `gold` → `expected_output`, `id`/`native_id` → `TestCase.id`, and `task_config` (`dataset_path`, `num_shots`, `primary_metric`, etc.) → `metadata`. On the results side, `task-*-predictions.jsonl` (per-instance `metrics`, `label`, model output) plus `metrics.json`'s `primary_score` map onto EvalPort's `ResultSet`, which already has a defined shape for per-case pass/fail plus aggregate metrics.
Two ways I could see this landing, no strong preference:
1. A standalone `olmes-openeval-adapter` package in the EvalPort repo, converting OLMES's `task-*-requests.jsonl`/`predictions.jsonl`/`metrics.json` outputs to/from EvalPort `Suite`/`ResultSet`. Zero footprint on this repo.
2. A small optional export utility inside `oe_eval/` if maintainers would rather it live here.
Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock. Let me know which direction you'd prefer, or if this isn't a fit for the roadmap right now — no worries either way.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
Start with OUTPUT_FORMATS.md and the existing oe_eval/ entry points to understand how task-*-requests.jsonl, task-*-predictions.jsonl, and metrics.json are produced. Compare the proposed standalone adapter and in-repository utility with EvalPort's actual JSON Schema; done means an agreed integration path and real conversion tests validating task instances, predictions, and aggregate metrics against that schema.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- python
- Área
- data, machine-learning
- Tipo de issue
- Nueva funcionalidad
- Dificultad
- 5/5
- Tiempo estimado
- Más de una semana
- Estado de actividad
- Activo
- Claridad
- Bastante claro
- Aptitud para principiantes
- 42/100