open-compass / open-compass/opencompass
[Bug] [Bug] LiveStemBench数据集评测时,推理结果正确。但是启动打分程序时报错 mmengine KeyError: '`cfg` or `default_args` must contain the key "type", but got {}\nNone'
@tonysy is already working on this.
Since Mar 28, 2025.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 869
- Avg merge
- 17h 52m
- Merged PRs (30d)
- 13
Description
先决条件
问题类型
我正在使用官方支持的任务/模型/数据集进行评估。
环境
opencompass最新版本
重现问题 - 代码/配置示例
from opencompass.models import VLLMwithChatTemplate
from mmengine.config import read_base
with read_base():
from opencompass.configs.datasets.ceval.ceval_gen import ceval_datasets
from opencompass.configs.datasets.cmmlu.cmmlu_gen import cmmlu_datasets
from opencompass.configs.datasets.IFEval.IFEval_gen import ifeval_datasets
from opencompass.configs.datasets.aime2024.aime2024_gen import aime2024_datasets
from opencompass.configs.datasets.livecodebench.livecodebench_gen import LCB_datasets
from opencompass.configs.datasets.livestembench.livestembench_gen import livestembench_datasets
datasets = [*ceval_datasets, *cmmlu_datasets, *ifeval_datasets, *aime2024_datasets,
*LCB_datasets, *livestembench_datasets]
models = [
dict(
type=VLLMwithChatTemplate,
abbr='qwq-32b',
path='./weights/QwQ-32B',
model_kwargs=dict(tensor_parallel_size=8, gpu_memory_utilization=0.8),
batch_size=16,
run_cfg=dict(num_gpus=8),
max_seq_len=32768,
max_out_len=8192,
generation_kwargs=dict(top_k=1, temperature=1e-6, top_p=0.9),
)
]
重现问题 - 命令或脚本
1、第一次启动评测,运行模式是infer,推理结果正常。生成的推理结果如下:
-rw-r--r-- 1 1000 1000 156190 Mar 26 13:08 LiveStemBench-biology_4.json
-rw-r--r-- 1 1000 1000 348126 Mar 26 11:57 LiveStemBench-biology_3.json
-rw-r--r-- 1 1000 1000 279707 Mar 26 10:53 LiveStemBench-biology_2.json
-rw-r--r-- 1 1000 1000 135478 Mar 26 09:45 LiveStemBench-biology_1.json
-rw-r--r-- 1 1000 1000 244885 Mar 26 08:36 LiveStemBench-biology_0.json
并且文件内容也完全正确。
2、第二次启动评测,模式选择eval。启动程序报错
File "opencompass/opencompass/utils/build.py", line 24, in build_model_from_cfg
return MODELS.build(model_cfg)
File "/usr/local/lib/python3.10/dist-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/usr/local/lib/python3.10/dist-packages/mmengine/registry/build_functions.py", line 72, in build_from_cfg
raise KeyError(
KeyError: 'cfg or default_args must contain the key "type", but got {}\nNone'
重现问题 - 错误信息
File "opencompass/opencompass/utils/build.py", line 24, in build_model_from_cfg
return MODELS.build(model_cfg)
File "/usr/local/lib/python3.10/dist-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/usr/local/lib/python3.10/dist-packages/mmengine/registry/build_functions.py", line 72, in build_from_cfg
raise KeyError(
KeyError: 'cfg or default_args must contain the key "type", but got {}\nNone'
其他信息
No response
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.