open-compass / open-compass/opencompass
[Bug] extract_non_reasoning_content error when predict is null
@acylam is already working on this.
Since Mar 20, 2025.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 869
- Avg merge
- 17h 52m
- Merged PRs (30d)
- 13
Description
Prerequisite
- I have searched Issues and Discussions but cannot get the expected help.
- The bug has not been fixed in the latest version.
Type
I'm evaluating with the officially supported tasks/models/datasets.
Environment
version 0.4.1
Reproduces the problem - code/configuration sample
from mmengine.config import read_base
from opencompass.models import OpenAISDK
with read_base():
from opencompass.configs.datasets.math.math_500_gen import math_datasets
# pip install math_verify latex2sympy2_extended
# pip install langdetect
# opencompass eval_gsm8k.py
# opencompass opencompass/eval_ds3.py
datasets = math_datasets
from opencompass.utils.text_postprocessors import extract_non_reasoning_content
models = [
dict(
abbr='deepseek-r1-h800-32768',
type=OpenAISDK,
path='/root/.cache/huggingface',
openai_api_base=['http://10.24.9.4:8000/v1'],
# tokenizer_path='/opt/weights/DeepSeek-V3',
key=
'token-abc123',
# meta_template=api_meta_template,
# query_per_second=2,
max_out_len=16384,
max_seq_len=32768, # 32768
temperature=0.6,
batch_size=16,
pred_postprocessor=dict(type=extract_non_reasoning_content),
retry=10
),
]
Reproduces the problem - command or script
test math-500 using deepseek-r1 model
Reproduces the problem - error message
INFO 03-20 14:38:29 init.py:190] Automatically detected platform cuda.
03/20 14:38:30 - OpenCompass - INFO - Try to load the data from /home/wzp/.cache/opencompass/./data/math/
Traceback (most recent call last):
File "/cx8k/fs100/wzp/code/opencompass/opencompass/tasks/openicl_eval.py", line 475, in
inferencer.run()
File "/cx8k/fs100/wzp/code/opencompass/opencompass/tasks/openicl_eval.py", line 86, in run
self._score()
File "/cx8k/fs100/wzp/code/opencompass/opencompass/tasks/openicl_eval.py", line 179, in _score
pred_strs = [proc(s, **kwargs) for s in pred_strs]
File "/cx8k/fs100/wzp/code/opencompass/opencompass/tasks/openicl_eval.py", line 179, in
pred_strs = [proc(s, **kwargs) for s in pred_strs]
File "/cx8k/fs100/wzp/code/opencompass/opencompass/utils/text_postprocessors.py", line 278, in extract_non_reasoning_content
if think_start_token not in text and think_end_token in text:
TypeError: argument of type 'NoneType' is not iterable
Other information
No response
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.