open-compass / open-compass/opencompass

[Bug] Unexpected version in summary, when using summarizers.leaderboard, for some datasets (chid, race, csl, eprstmt).

Open
#760 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.5k
Forks
869
Avg merge
17h 52m
Merged PRs (30d)
13

Description

Prerequisite
Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

opencompass verison: v0.2.0
data source: OpenCompassData-complete-20231110

Reproduces the problem - code/configuration sample

Part of my config file (only show the datasets related to this bug):

from mmengine.config import read_base

with read_base():
    from .summarizers.leaderboard import summarizer

    from .datasets.FewCLUE_chid.FewCLUE_chid_ppl_8f2872 import chid_datasets
    from .datasets.race.race_ppl_5831a0 import race_datasets
    from .datasets.FewCLUE_csl.FewCLUE_csl_ppl_841b62 import csl_datasets
    from .datasets.FewCLUE_eprstmt.FewCLUE_eprstmt_ppl_f1e631 import eprstmt_datasets

datasets = sum((v for k, v in locals().items() if k.endswith('_datasets')), [])
Reproduces the problem - command or script

python run.py configs/myeval.py

Reproduces the problem - error message

No error message.

Other information

Part of the summary_[timestamp.txt] (only show the datasets related to this bug):

dataset                                 version    metric            mode  
--------------------------------------  ---------  ----------------  ------
--------- 考试 Exam ---------           -          -                 -      

--------- 语言 Language ---------       -          -                 -      
chid-dev                                90451d     accuracy          ppl    
--------- 知识 Knowledge ---------      -          -                 -      

--------- 理解 Understanding ---------  -          -                 -      
race-middle                             dda65f     accuracy          ppl    
race-high                               dda65f     accuracy          ppl    
csl_dev                                 46f772     accuracy          ppl    
eprstmt-dev                             ca49a2     accuracy          ppl    
--------- 推理 Reasoning ---------      -          -                 -     

These datasets's version in summary is not same as their version used in the config file, while other datasets's version is.
And the versions above are not appear in the corresponding configs/datasets/[dataset], for example, there is no "race_ppl_5831a0.py" in "configs/datasets/race".

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the summarizers/leaderboard configuration imported by configs/myeval.py, then trace how the dataset version is obtained when running python run.py configs/myeval.py. Compare that path with the dataset configurations for FewCLUE_chid, race, FewCLUE_csl, and FewCLUE_eprstmt; done means the summary reports the versions selected by those configurations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.