open-compass / open-compass/opencompass

[Bug] 评测mbpp数据集时,infer过程报错TypeError: can only concatenate tuple (not "str") to tuple

Open
#908 0 comments 0 reactions 1 assignee View on GitHub

@kennymckormick is already working on this.

Since Feb 22, 2024.

Dominant language
Python
Stars
7.5k
Forks
869
Avg merge
17h 52m
Merged PRs (30d)
13

Description

Prerequisite
Type

I'm evaluating with the officially supported tasks/models/datasets.

Environment

1

Reproduces the problem - code/configuration sample

1

Reproduces the problem - command or script

1

Reproduces the problem - error message

Traceback (most recent call last):
File "/home/dengkaibiao/opencompass/opencompass/tasks/openicl_infer.py", line 153, in
inferencer.run()
File "/home/dengkaibiao/opencompass/opencompass/tasks/openicl_infer.py", line 71, in run
self.dataset = build_dataset_from_cfg(self.dataset_cfg)
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/opencompass/utils/build.py", line 13, in build_dataset_from_cfg
return LOAD_DATASET.build(dataset_cfg)
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/opencompass/datasets/base.py", line 12, in init
self.dataset = self.load(**kwargs)
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/opencompass/datasets/mbpp.py", line 35, in load
train = load_dataset('json', data_files=path,
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/load.py", line 2106, in load_dataset
builder_instance = load_dataset_builder(
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/load.py", line 1792, in load_dataset_builder
dataset_module = dataset_module_factory(
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/load.py", line 1410, in dataset_module_factory
).get_module()
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/load.py", line 947, in get_module
data_files = DataFilesDict.from_patterns(
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/data_files.py", line 671, in from_patterns
DataFilesList.from_patterns(
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/data_files.py", line 577, in from_patterns
resolve_pattern(
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/data_files.py", line 335, in resolve_pattern
protocol_prefix = fs.protocol + "://" if fs.protocol != "file" else ""
TypeError: can only concatenate tuple (not "str") to tuple

Other information

配置文件信息config/eval_deng.py:

from mmengine.config import read_base
from opencompass.models import OpenAI
from opencompass.partitioners import NaivePartitioner
from opencompass.runners import LocalRunner
from opencompass.tasks import OpenICLInferTask

with read_base():
from .datasets.collections.chat_medium import datasets
from .summarizers.medium import summarizer
from .datasets.mbpp.mbpp_gen import mbpp_datasets

api_meta_template = dict(
round=[
dict(role='HUMAN', api_role='HUMAN'),
dict(role='BOT', api_role='BOT', generate=True),
],
)

datasets = [*mbpp_datasets]

models = [
dict(abbr='deng',
type=OpenAI, path='deng',
key='http://127.0.0.1:1000', # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
meta_template=api_meta_template,
query_per_second=1,
max_out_len=4096, max_seq_len=4096, batch_size=8),
]

infer = dict(
partitioner=dict(type=NaivePartitioner),
runner=dict(
type=LocalRunner,
max_num_workers=1,
task=dict(type=OpenICLInferTask)),
)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.