open-compass / open-compass/opencompass
[Bug] 评测mbpp数据集时,infer过程报错TypeError: can only concatenate tuple (not "str") to tuple
@kennymckormick is already working on this.
Since Feb 22, 2024.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 869
- Avg merge
- 17h 52m
- Merged PRs (30d)
- 13
Description
Prerequisite
- I have searched Issues and Discussions but cannot get the expected help.
- The bug has not been fixed in the latest version.
Type
I'm evaluating with the officially supported tasks/models/datasets.
Environment
1
Reproduces the problem - code/configuration sample
1
Reproduces the problem - command or script
1
Reproduces the problem - error message
Traceback (most recent call last):
File "/home/dengkaibiao/opencompass/opencompass/tasks/openicl_infer.py", line 153, in
inferencer.run()
File "/home/dengkaibiao/opencompass/opencompass/tasks/openicl_infer.py", line 71, in run
self.dataset = build_dataset_from_cfg(self.dataset_cfg)
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/opencompass/utils/build.py", line 13, in build_dataset_from_cfg
return LOAD_DATASET.build(dataset_cfg)
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/opencompass/datasets/base.py", line 12, in init
self.dataset = self.load(**kwargs)
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/opencompass/datasets/mbpp.py", line 35, in load
train = load_dataset('json', data_files=path,
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/load.py", line 2106, in load_dataset
builder_instance = load_dataset_builder(
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/load.py", line 1792, in load_dataset_builder
dataset_module = dataset_module_factory(
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/load.py", line 1410, in dataset_module_factory
).get_module()
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/load.py", line 947, in get_module
data_files = DataFilesDict.from_patterns(
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/data_files.py", line 671, in from_patterns
DataFilesList.from_patterns(
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/data_files.py", line 577, in from_patterns
resolve_pattern(
File "/home/dengkaibiao/anaconda3/lib/python3.10/site-packages/datasets/data_files.py", line 335, in resolve_pattern
protocol_prefix = fs.protocol + "://" if fs.protocol != "file" else ""
TypeError: can only concatenate tuple (not "str") to tuple
Other information
配置文件信息config/eval_deng.py:
from mmengine.config import read_base
from opencompass.models import OpenAI
from opencompass.partitioners import NaivePartitioner
from opencompass.runners import LocalRunner
from opencompass.tasks import OpenICLInferTask
with read_base():
from .datasets.collections.chat_medium import datasets
from .summarizers.medium import summarizer
from .datasets.mbpp.mbpp_gen import mbpp_datasets
api_meta_template = dict(
round=[
dict(role='HUMAN', api_role='HUMAN'),
dict(role='BOT', api_role='BOT', generate=True),
],
)
datasets = [*mbpp_datasets]
models = [
dict(abbr='deng',
type=OpenAI, path='deng',
key='http://127.0.0.1:1000', # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
meta_template=api_meta_template,
query_per_second=1,
max_out_len=4096, max_seq_len=4096, batch_size=8),
]
infer = dict(
partitioner=dict(type=NaivePartitioner),
runner=dict(
type=LocalRunner,
max_num_workers=1,
task=dict(type=OpenICLInferTask)),
)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.