open-compass / open-compass/opencompass

[Bug] [Bug] LiveStemBench数据集评测时,推理结果正确。但是启动打分程序时报错 mmengine KeyError: '`cfg` or `default_args` must contain the key "type", but got {}\nNone'

Open
#1,980 1 comment 0 reactions 1 assignee View on GitHub

@tonysy is already working on this.

Since Mar 28, 2025.

Dominant language
Python
Stars
7.5k
Forks
869
Avg merge
17h 52m
Merged PRs (30d)
13

Description

先决条件
问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

环境

opencompass最新版本

重现问题 - 代码/配置示例

from opencompass.models import VLLMwithChatTemplate
from mmengine.config import read_base

with read_base():
from opencompass.configs.datasets.ceval.ceval_gen import ceval_datasets
from opencompass.configs.datasets.cmmlu.cmmlu_gen import cmmlu_datasets
from opencompass.configs.datasets.IFEval.IFEval_gen import ifeval_datasets
from opencompass.configs.datasets.aime2024.aime2024_gen import aime2024_datasets
from opencompass.configs.datasets.livecodebench.livecodebench_gen import LCB_datasets
from opencompass.configs.datasets.livestembench.livestembench_gen import livestembench_datasets

datasets = [*ceval_datasets, *cmmlu_datasets, *ifeval_datasets, *aime2024_datasets,
*LCB_datasets, *livestembench_datasets]

models = [
dict(
type=VLLMwithChatTemplate,
abbr='qwq-32b',
path='./weights/QwQ-32B',
model_kwargs=dict(tensor_parallel_size=8, gpu_memory_utilization=0.8),
batch_size=16,
run_cfg=dict(num_gpus=8),
max_seq_len=32768,
max_out_len=8192,
generation_kwargs=dict(top_k=1, temperature=1e-6, top_p=0.9),
)
]

重现问题 - 命令或脚本

1、第一次启动评测,运行模式是infer,推理结果正常。生成的推理结果如下:
-rw-r--r-- 1 1000 1000 156190 Mar 26 13:08 LiveStemBench-biology_4.json
-rw-r--r-- 1 1000 1000 348126 Mar 26 11:57 LiveStemBench-biology_3.json
-rw-r--r-- 1 1000 1000 279707 Mar 26 10:53 LiveStemBench-biology_2.json
-rw-r--r-- 1 1000 1000 135478 Mar 26 09:45 LiveStemBench-biology_1.json
-rw-r--r-- 1 1000 1000 244885 Mar 26 08:36 LiveStemBench-biology_0.json
并且文件内容也完全正确。

2、第二次启动评测,模式选择eval。启动程序报错

File "opencompass/opencompass/utils/build.py", line 24, in build_model_from_cfg
return MODELS.build(model_cfg)
File "/usr/local/lib/python3.10/dist-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/usr/local/lib/python3.10/dist-packages/mmengine/registry/build_functions.py", line 72, in build_from_cfg
raise KeyError(
KeyError: 'cfg or default_args must contain the key "type", but got {}\nNone'

重现问题 - 错误信息

File "opencompass/opencompass/utils/build.py", line 24, in build_model_from_cfg
return MODELS.build(model_cfg)
File "/usr/local/lib/python3.10/dist-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/usr/local/lib/python3.10/dist-packages/mmengine/registry/build_functions.py", line 72, in build_from_cfg
raise KeyError(
KeyError: 'cfg or default_args must contain the key "type", but got {}\nNone'

其他信息

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.