modelgauge.prompt_pipeline should accept column headers as they appear in official prompt set files
@rogthefrog is already working on this.
Since Mar 20, 2025.
Assessment
This issue has not been assessed yet.
Description
The prompt_pipelines module expects prompt files to have PROMPT_CSV_INPUT_COLUMNS = ["UID", "Text"] as headers. The prompt set files use different column names, so you can't use prompt set files as inputs to modelgauge as-is.
Fix that.
Also document the acceptable column names.
Here's the prompt schema file Kurt created:
https://github.com/mlcommons/sugar/blob/prompts/prompts/schema.sql
- Dominant language
- Python
- Stars
- 134
- Forks
- 36
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 17
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from mlcommons/modelbench
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
mlcommons/modelbench#1659 · 1 comment ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
mlcommons/modelbench#1656 ·
-
Difficulty 2/5 Half a day Newbie friendliness 84/100
mlcommons/modelbench#1629 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 45/100
mlcommons/modelbench#1660 ·
-
Difficulty 3/5 1-2 days Newbie friendliness 45/100
mlcommons/modelbench#1658 ·
All issues in mlcommons/modelbench
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100