open-compass / open-compass/opencompass
[Bug] Long text evaluation parameters are not clear
@liushz is already working on this.
Since Apr 10, 2024.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 869
- Avg merge
- 17h 52m
- Merged PRs (30d)
- 13
Description
Prerequisite
- I have searched Issues and Discussions but cannot get the expected help.
- The bug has not been fixed in the latest version.
Type
I'm evaluating with the officially supported tasks/models/datasets.
Environment
python 3.10.1
OpenCompass 0.2.3
vllm 0.2.3
Reproduces the problem - code/configuration sample
configs/models/chatglm/vllm_chatglm2_6b_32k.py
from opencompass.models import VLLM
models = [
dict(
type=VLLM,
abbr='chatglm2-6b-32k-vllm',
path='THUDM/chatglm2-6b-32k',
max_out_len=512,
max_seq_len=4096,
batch_size=32,
generation_kwargs=dict(temperature=0),
run_cfg=dict(num_gpus=1, num_procs=1),
)
]
Reproduces the problem - command or script
python run.py --model vllm_chatglm2_6b_32k --datasets longbench leval
Reproduces the problem - error message
The difference between the evaluation result parameters and the document long text evaluation is about 20 points, The score for the document can not be reproduced.
- “max_seq_len、max_out_len” Should these two parameters be modified in any way?
Other information
No response
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.