open-compass / open-compass/opencompass
[Feature] AGIEval数据集是否可以支持few-shot评测设置?
Open
@bittersweet1999 is already working on this.
Since May 14, 2024.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 869
- Avg merge
- 17h 52m
- Merged PRs (30d)
- 13
Description
描述该功能
opencompass/opencompass/datasets/agieval/agieval.py 文件显示,目前AGIEval评测仅支持zero-shot设置:
def load(path: str, name: str, setting_name: str):
from .dataset_loader import load_dataset, load_dataset_as_result_schema
assert setting_name in 'zero-shot', 'only support zero-shot setting'
dataset_wo_label = load_dataset(name, setting_name, path)
dataset_with_label = load_dataset_as_result_schema(name, path)
dataset = []
for d1, d2 in zip(dataset_wo_label, dataset_with_label):
dataset.append({
'id': d2.index,
'problem_input': d1['context'],
'label': d2.label,
})
dataset = Dataset.from_list(dataset)
return dataset
原生AGIEval支持few-shot设置,是否可以追加few-shot评测设置?
是否希望自己实现该功能?
- 我希望自己来实现这一功能,并向 OpenCompass 贡献代码!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.