bigscience-workshop / bigscience-workshop/promptsource
Optional EvalPort export for applied P3 prompt templates
- Dominant language
- Python
- Stars
- 3k
- Forks
- 375
- PR merge metrics
- No merged PRs in 30d
Description
Hi PromptSource team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open, framework-agnostic JSON spec for portable LLM eval datasets (Apache 2.0, a `TestCase`/`Suite` schema with a real JSON Schema validator, so a dataset can move between eval tools without a bespoke converter).
I read the real template format rather than guessing. `promptsource/templates/ag_news/templates.yaml`:
```yaml
24e44a81-a18a-42dd-a71c-5b31b2d2cb39: !Template
answer_choices: World politics ||| Sports ||| Business ||| Science and technology
id: 24e44a81-a18a-42dd-a71c-5b31b2d2cb39
jinja: "What label best describes this news article?\n{{text}} ||| \n{{answer_choices[label] }}"
metadata: !TemplateMetadata
metrics:
- Accuracy
name: classify_question_first
```
and CONTRIBUTING.md confirms the contract: "a template must produce two strings: an input and a target," separated by `|||`, with `answer_choices` and `metadata.metrics` (e.g. `Accuracy`) as first-class fields.
That's a very close match to an EvalPort `TestCase`: the rendered input string → `TestCase.input`, the rendered target string → `expected_output`, `answer_choices` → the option set, and `metadata.metrics` → the EvalPort `Grader` type (`exact_match` for Accuracy, etc.). Applying a `DatasetTemplates` collection to its underlying HF dataset split would produce a ready-made EvalPort `Suite`.
Would it be of interest to have a small `promptsource-openeval-adapter` in the EvalPort repo that walks a `DatasetTemplates`/`Template.apply()` output into an EvalPort `Suite`, per dataset/subset, with attribution to P3 and this repo's license preserved? I'd do the work and validate against EvalPort's real schema — filing this as an issue first per CONTRIBUTING.md norms rather than sending a PR out of nowhere. No changes needed in this repo either way. Let me know if this is of interest, or not — no worries either way.
Contributor guide
Research direction
Start with promptsource/templates/ag_news/templates.yaml and CONTRIBUTING.md, then inspect DatasetTemplates and Template.apply() to verify the stated input/target contract. Validate any proposed adapter against EvalPort’s real JSON schema, with per-dataset or subset Suite output and preserved attribution and licensing as the done criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100