open-compass / open-compass/opencompass

[Bug] Ceval_gen not compare for modelscope

Open
#2,003 1 comment 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.5k
Forks
869
Avg merge
17h 52m
Merged PRs (30d)
13

Description

Prerequisite
Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

common

Reproduces the problem - code/configuration sample

python run.py --datasets ceval_gen_2daf24 --hf-path /media/fangzhen/DATA/zth/2025-04-train-LLM/LLaMA-Factory-main/saves/Qwen-1.5B/pt --debug --hf-type chat

Reproduces the problem - command or script

python run.py --datasets ceval_gen_2daf24 --hf-path /media/fangzhen/DATA/zth/2025-04-train-LLM/LLaMA-Factory-main/saves/Qwen-1.5B/pt --debug --hf-type chat

Reproduces the problem - error message

04/04 16:30:38 - OpenCompass - INFO - Loading ceval_gen_2daf24: /media/fangzhen/DATA/zth/2025-04-train-LLM/opencompass/opencompass/configs/./datasets/ceval/ceval_gen_2daf24.py
04/04 16:30:38 - OpenCompass - INFO - Loading example: /media/fangzhen/DATA/zth/2025-04-train-LLM/opencompass/opencompass/configs/./summarizers/example.py
04/04 16:30:38 - OpenCompass - INFO - Current exp folder: outputs/default/20250404_163038
04/04 16:30:38 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
Traceback (most recent call last):
File "/home/fangzhen/anaconda3/envs/opencompass/lib/python3.10/site-packages/modelscope/utils/import_utils.py", line 462, in _get_module
return importlib.import_module('.' + module_name, self.name)
File "/home/fangzhen/anaconda3/envs/opencompass/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/home/fangzhen/anaconda3/envs/opencompass/lib/python3.10/site-packages/modelscope/msdatasets/init.py", line 2, in
from modelscope.msdatasets.ms_dataset import MsDataset
File "/home/fangzhen/anaconda3/envs/opencompass/lib/python3.10/site-packages/modelscope/msdatasets/ms_dataset.py", line 25, in
from modelscope.msdatasets.utils.hf_datasets_util import load_dataset_with_ctx
File "/home/fangzhen/anaconda3/envs/opencompass/lib/python3.10/site-packages/modelscope/msdatasets/utils/hf_datasets_util.py", line 19, in
from datasets.data_files import (
ImportError: cannot import name 'get_metadata_patterns' from 'datasets.data_files' (/home/fangzhen/anaconda3/envs/opencompass/lib/python3.10/site-packages/datasets/data_files.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/media/fangzhen/DATA/zth/2025-04-train-LLM/opencompass/run.py", line 4, in
main()
File "/media/fangzhen/DATA/zth/2025-04-train-LLM/opencompass/opencompass/cli/main.py", line 328, in main
tasks = partitioner(cfg)
File "/media/fangzhen/DATA/zth/2025-04-train-LLM/opencompass/opencompass/partitioners/base.py", line 83, in call
tasks = self.partition(**model_and_dataset_args,
File "/media/fangzhen/DATA/zth/2025-04-train-LLM/opencompass/opencompass/partitioners/num_worker.py", line 69, in partition
dataset_size = self.get_size(dataset)
File "/media/fangzhen/DATA/zth/2025-04-train-LLM/opencompass/opencompass/partitioners/num_worker.py", line 141, in get_size
dataset = build_dataset_from_cfg(dataset)
File "/media/fangzhen/DATA/zth/2025-04-train-LLM/opencompass/opencompass/utils/build.py", line 12, in build_dataset_from_cfg
return LOAD_DATASET.build(dataset_cfg)
File "/home/fangzhen/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/home/fangzhen/anaconda3/envs/opencompass/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/media/fangzhen/DATA/zth/2025-04-train-LLM/opencompass/opencompass/datasets/base.py", line 17, in init
dataset = self.load(**kwargs)
File "/media/fangzhen/DATA/zth/2025-04-train-LLM/opencompass/opencompass/datasets/ceval.py", line 22, in load
from modelscope import MsDataset
File "", line 1075, in _handle_fromlist
File "/home/fangzhen/anaconda3/envs/opencompass/lib/python3.10/site-packages/modelscope/utils/import_utils.py", line 437, in getattr
module = self._get_module(self._class_to_module[name])
File "/home/fangzhen/anaconda3/envs/opencompass/lib/python3.10/site-packages/modelscope/utils/import_utils.py", line 464, in _get_module
raise RuntimeError(
RuntimeError: Failed to import modelscope.msdatasets because of the following error (look up to see its traceback):
cannot import name 'get_metadata_patterns' from 'datasets.data_files' (/home/fangzhen/anaconda3/envs/opencompass/lib/python3.10/site-packages/datasets/data_files.py)

Other information

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.