allenai / allenai/olmo-eval

Task spec API is ambiguous; propose canonical <dataset>:<format>:<metric> schema

Aperta
#82 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
70
Fork
21
Merge medio
2g 11h
PR unite (30g)
41

Descrizione

## Problem
Task specs are currently composable colon-delimited variants (`task[:variant1[:variant2...]]`), but the semantics of each segment are not explicit or enforced.

Concrete example from current quickstart docs:
- `humaneval:3shot` conveys prompt format / few-shot regime, but not metric.
- `humaneval:bpb` conveys metric, but not format/regime.

## Proposal
Adopt an explicit minimal schema for task specs and apply it consistently. For example:

`::`

Examples:
- `humaneval:3shot:bpb`
- `humaneval:0shot:pass_at_1`

There's an open question, of course, for what `mc|rc` count as vs `0shot|3shot|8shot` count as

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.