open-compass / open-compass/opencompass
[Bug] Unexpected version in summary, when using summarizers.leaderboard, for some datasets (chid, race, csl, eprstmt).
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 869
- Avg merge
- 17h 52m
- Merged PRs (30d)
- 13
Description
Prerequisite
- I have searched Issues and Discussions but cannot get the expected help.
- The bug has not been fixed in the latest version.
Type
I'm evaluating with the officially supported tasks/models/datasets.
Environment
opencompass verison: v0.2.0
data source: OpenCompassData-complete-20231110
Reproduces the problem - code/configuration sample
Part of my config file (only show the datasets related to this bug):
from mmengine.config import read_base
with read_base():
from .summarizers.leaderboard import summarizer
from .datasets.FewCLUE_chid.FewCLUE_chid_ppl_8f2872 import chid_datasets
from .datasets.race.race_ppl_5831a0 import race_datasets
from .datasets.FewCLUE_csl.FewCLUE_csl_ppl_841b62 import csl_datasets
from .datasets.FewCLUE_eprstmt.FewCLUE_eprstmt_ppl_f1e631 import eprstmt_datasets
datasets = sum((v for k, v in locals().items() if k.endswith('_datasets')), [])
Reproduces the problem - command or script
python run.py configs/myeval.py
Reproduces the problem - error message
No error message.
Other information
Part of the summary_[timestamp.txt] (only show the datasets related to this bug):
dataset version metric mode
-------------------------------------- --------- ---------------- ------
--------- 考试 Exam --------- - - -
--------- 语言 Language --------- - - -
chid-dev 90451d accuracy ppl
--------- 知识 Knowledge --------- - - -
--------- 理解 Understanding --------- - - -
race-middle dda65f accuracy ppl
race-high dda65f accuracy ppl
csl_dev 46f772 accuracy ppl
eprstmt-dev ca49a2 accuracy ppl
--------- 推理 Reasoning --------- - - -
These datasets's version in summary is not same as their version used in the config file, while other datasets's version is.
And the versions above are not appear in the corresponding configs/datasets/[dataset], for example, there is no "race_ppl_5831a0.py" in "configs/datasets/race".
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the summarizers/leaderboard configuration imported by configs/myeval.py, then trace how the dataset version is obtained when running python run.py configs/myeval.py. Compare that path with the dataset configurations for FewCLUE_chid, race, FewCLUE_csl, and FewCLUE_eprstmt; done means the summary reports the versions selected by those configurations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100