bigscience-workshop / bigscience-workshop/promptsource

Optional EvalPort export for applied P3 prompt templates

Open
#860 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3k
Forks
375
PR merge metrics
No merged PRs in 30d

Description

Hi PromptSource team — I maintain [EvalPort](https://github.com/adhabnr-ux/evalport), an open, framework-agnostic JSON spec for portable LLM eval datasets (Apache 2.0, a `TestCase`/`Suite` schema with a real JSON Schema validator, so a dataset can move between eval tools without a bespoke converter).

I read the real template format rather than guessing. `promptsource/templates/ag_news/templates.yaml`:

```yaml
24e44a81-a18a-42dd-a71c-5b31b2d2cb39: !Template
answer_choices: World politics ||| Sports ||| Business ||| Science and technology
id: 24e44a81-a18a-42dd-a71c-5b31b2d2cb39
jinja: "What label best describes this news article?\n{{text}} ||| \n{{answer_choices[label] }}"
metadata: !TemplateMetadata
metrics:
- Accuracy
name: classify_question_first
```

and CONTRIBUTING.md confirms the contract: "a template must produce two strings: an input and a target," separated by `|||`, with `answer_choices` and `metadata.metrics` (e.g. `Accuracy`) as first-class fields.

That's a very close match to an EvalPort `TestCase`: the rendered input string → `TestCase.input`, the rendered target string → `expected_output`, `answer_choices` → the option set, and `metadata.metrics` → the EvalPort `Grader` type (`exact_match` for Accuracy, etc.). Applying a `DatasetTemplates` collection to its underlying HF dataset split would produce a ready-made EvalPort `Suite`.

Would it be of interest to have a small `promptsource-openeval-adapter` in the EvalPort repo that walks a `DatasetTemplates`/`Template.apply()` output into an EvalPort `Suite`, per dataset/subset, with attribution to P3 and this repo's license preserved? I'd do the work and validate against EvalPort's real schema — filing this as an issue first per CONTRIBUTING.md norms rather than sending a PR out of nowhere. No changes needed in this repo either way. Let me know if this is of interest, or not — no worries either way.

Contributor guide

Open the contributing guide

Research direction

Start with promptsource/templates/ag_news/templates.yaml and CONTRIBUTING.md, then inspect DatasetTemplates and Template.apply() to verify the stated input/target contract. Validate any proposed adapter against EvalPort’s real JSON schema, with per-dataset or subset Suite output and preserved attribution and licensing as the done criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.