open-compass / open-compass/opencompass
[Bug] CLUEWSC的测试结果全是 50%
Open
@bittersweet1999 is already working on this.
Since May 21, 2024.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 869
- Avg merge
- 17h 52m
- Merged PRs (30d)
- 13
Description
Prerequisite
- I have searched Issues and Discussions but cannot get the expected help.
- The bug has not been fixed in the latest version.
Type
I'm evaluating with the officially supported tasks/models/datasets.
Environment
使用的版本是 0.2.3
Reproduces the problem - code/configuration sample
python3 ${OPENCOMPASS_PATH}/run.py --work-dir ${OPENCOMPASS_PATH} \
--datasets FewCLUE_cluewsc_ppl \
--hf-path ${model_path} \
--config-dir ${OPENCOMPASS_PATH}/configs \
--model-kwargs device_map='auto' trust_remote_code=True \
--tokenizer-kwargs padding_side='right' truncation='left' trust_remote_code=True add_special_tokens=False \
--max-seq-len 4096 \
--max-out-len 100 \
--batch-size 1 \
--reuse results/${model_name} \
--num-gpus 8
Reproduces the problem - command or script
python3 ${OPENCOMPASS_PATH}/run.py --work-dir ${OPENCOMPASS_PATH} \
--datasets FewCLUE_cluewsc_ppl \
--hf-path ${model_path} \
--config-dir ${OPENCOMPASS_PATH}/configs \
--model-kwargs device_map='auto' trust_remote_code=True \
--tokenizer-kwargs padding_side='right' truncation='left' trust_remote_code=True add_special_tokens=False \
--max-seq-len 4096 \
--max-out-len 100 \
--batch-size 1 \
--reuse results/${model_name} \
--num-gpus 8
Reproduces the problem - error message
我测试了 Qwen1.5-1.8B, Gemma-2b, rwkv-5-world-1b5 和 MiniCPM-2B-sft-fp32。结果是:
50.31/50.31/50.31/52.20
这个结果和瞎猜没区别。而且这个任务不能选择 few-shot,我理解应该任意任务都可以设置 few-shot 而不是像现在这种每个任务是写死的?
Other information
No response
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.