Task spec API is ambiguous; propose canonical <dataset>:<format>:<metric> schema
- Lingua principale
- Python
- Stelle
- 70
- Fork
- 21
- Merge medio
- 2g 11h
- PR unite (30g)
- 41
Descrizione
## Problem
Task specs are currently composable colon-delimited variants (`task[:variant1[:variant2...]]`), but the semantics of each segment are not explicit or enforced.
Concrete example from current quickstart docs:
- `humaneval:3shot` conveys prompt format / few-shot regime, but not metric.
- `humaneval:bpb` conveys metric, but not format/regime.
## Proposal
Adopt an explicit minimal schema for task specs and apply it consistently. For example:
`::`
Examples:
- `humaneval:3shot:bpb`
- `humaneval:0shot:pass_at_1`
There's an open question, of course, for what `mc|rc` count as vs `0shot|3shot|8shot` count as
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.