modelscope / modelscope/evalscope
Question: standard vs CoT prompting in API calls
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 489
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 87
Description
Hi, is there currently a way to evaluate the same dataset using different prompt templates (e.g., MultipleChoiceTemplate.SINGLE_ANSWER and MultipleChoiceTemplate.SINGLE_ANSWER_COT) via the API or CLI call, without redefining the benchmark adapter?
This would make it much easier to compare LLMs’ performance on standard vs. CoT prompting styles using the same dataset and evaluation logic.
Warm regards,
Yang
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the API and CLI entry points for benchmark evaluation and how benchmark adapters select prompt templates. Check whether MultipleChoiceTemplate.SINGLE_ANSWER and SINGLE_ANSWER_COT can be selected without redefining an adapter. Done means the same dataset and evaluation logic can run with both templates through a documented API or CLI option.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, cli, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100