GRPO多轮训练rollout输出<im_end>，之后修正该条rollout将会报错

Describe the bug
What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程，最好有截图)

GRPO多轮训练rollout偶尔会输出空字符串，之后修正该条rollout后将会报错

scheduler代码：

def step(self, infer_request, response_choice, current_turn):
 
     completion = response_choice.message.content
     if completion == '':
         print('finished by zero output')
         infer_request.messages.append({"role": "user", "content": 'please continue answer the question'})
         return {
         'infer_request': infer_request,
         'rollout_infos': {
             'code_results': 'not put',
             'num_turns': current_turn,
         }
     }
     # print('------------------------')
     # print(current_turn, completion)
     # print('------------------------')
     token_ids = response_choice.token_ids
     loss_mask = [1] * len(token_ids)

     code = self._extract_code(completion)
     code_results = self._repl_code_run(self.code_env, code)
     
     results_with_tokens = f'\n<code_result>\n{code_results}\n</code_result>\n'
     # infer_request.messages[-1]['content'] += (results_with_tokens)
     infer_request.messages.append({"role": "user", "content": results_with_tokens})
     # # print(infer_request)

     # # print(self.infer_engine)
     # tokenizer = self.infer_engine.default_template.tokenizer
     # result_tokens = tokenizer.encode(results_with_tokens, add_special_tokens=False)
     # token_ids.extend(result_tokens)
     # loss_mask.extend([0] * len(result_tokens))

     return {
         'infer_request': infer_request,
         'rollout_infos': {
             'code_results': results_with_tokens,
             'num_turns': current_turn,
         }
     }

CUDA_VISIBLE_DEVICES=5,6
swift rollout
--model /ssd/yuky/TwT/src/output_dir/stage2-outputdir/v4-20251222-134020/checkpoint-696
--external_plugins /ssd/yuky/TwT/src/rl/myrlhf/ms-swift/examples/train/grpo/plugin/twtplugin.py
--multi_turn_scheduler twt_scheduler
--vllm_gpu_memory_utilization 0.8
--max_turns 10
# --stop_words ''

NCCL_DEBUG=INFO \

REWARD_API_ADDRESS=0.0.0.0 \

QWEN_API_PORT=8000 \

VQA_WEIGHT=1 \

FMT_WEIGHT=0.5 \

CODE_WEIGHT=0.1 \

MAX_PIXELS=3211264 \

CUDA_VISIBLE_DEVICES=0,1,2,3,4,6,7
deepspeed --hostfile=hostfile_5655_1
scripts/rlhf_ds.py
--rlhf_type grpo
--model /ssd/yuky/TwT/src/output_dir/stage2-outputdir/v4-20251222-134020/checkpoint-696
--external_plugins /ssd/yuky/TwT/src/rl/myrlhf/ms-swift/examples/train/grpo/plugin/twtplugin.py
--reward_funcs re_format
--use_vllm true
--vllm_mode server
--vllm_server_host 0.0.0.0
--vllm_server_port 8000
--vllm_server_timeout 60
--train_type full
--torch_dtype bfloat16
--dataset /ssd/yuky/TwT/data/multimodalQA-used/pawpularity.json
--dataset_shuffle true
--train_dataloader_shuffle true
--max_pixels 2408448
--max_length 10240
--max_completion_length 10240
--freeze_aligner false
--stop_words '' ''
--num_train_epochs 2
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--padding_side left
--learning_rate 5e-7
--lr_scheduler_type cosine_with_min_lr
--lr_scheduler_kwargs '{"min_lr_rate": 0.1, "num_cycles": 0.5}'
--gradient_accumulation_steps 2
--save_strategy 'steps'
--eval_strategy 'no'
--split_dataset_ratio 0
--eval_steps 20000
--save_steps 100
--save_total_limit 100000
--logging_steps 1
--output_dir output_dir
--warmup_ratio 0.03
--dataloader_num_workers 8
--num_generations 2
--temperature 1.0
--beta 0.01
--top_p 0.9
--top_k 50
--repetition_penalty 1.05
--deepspeed zero3
--log_completions true
--report_to tensorboard
--async_generate false
--num_iterations 1
--overlong_filter true
--offload_optimizer true
--offload_model true
--attn_impl flash_attn
--vllm_enforce_eager true \

报错信息

Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等)

CUDA 12.8

Name Version Build Channel

_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
absl-py 2.3.1 pypi_0 pypi
accelerate 1.12.0 pypi_0 pypi
addict 2.4.0 pypi_0 pypi
aiofiles 24.1.0 pypi_0 pypi
aiohappyeyeballs 2.6.1 pypi_0 pypi
aiohttp 3.13.2 pypi_0 pypi
aiosignal 1.4.0 pypi_0 pypi
airportsdata 20250909 pypi_0 pypi
alembic 1.17.2 pypi_0 pypi
aliyun-python-sdk-core 2.16.0 pypi_0 pypi
aliyun-python-sdk-kms 2.16.5 pypi_0 pypi
annotated-doc 0.0.4 pypi_0 pypi
annotated-types 0.7.0 pypi_0 pypi
anthropic 0.75.0 pypi_0 pypi
antlr4-python3-runtime 4.9.3 pypi_0 pypi
anyio 4.11.0 pypi_0 pypi
apache-tvm-ffi 0.1.3 pypi_0 pypi
astor 0.8.1 pypi_0 pypi
asttokens 3.0.1 pyhd8ed1ab_0 conda-forge
async-timeout 5.0.1 pypi_0 pypi
attrdict 2.0.1 pypi_0 pypi
attrs 25.4.0 pypi_0 pypi
audioread 3.1.0 pypi_0 pypi
auto-gptq 0.7.1 pypi_0 pypi
autoawq 0.2.9 pypi_0 pypi
autopep8 2.3.2 pypi_0 pypi
av 16.0.1 pypi_0 pypi
binpacking 1.5.2 pypi_0 pypi
bitsandbytes 0.48.2 pypi_0 pypi
blake3 1.0.8 pypi_0 pypi
bleach 6.3.0 pypi_0 pypi
blobfile 3.0.0 pypi_0 pypi
brotli 1.2.0 pypi_0 pypi
build 1.3.0 pypi_0 pypi
bzip2 1.0.8 h5eee18b_6
ca-certificates 2025.11.12 hbd8a1cb_0 conda-forge
cachetools 6.2.2 pypi_0 pypi
catboost 1.2.8 pypi_0 pypi
cbor2 5.7.1 pypi_0 pypi
certifi 2025.11.12 pypi_0 pypi
cffi 2.0.0 pypi_0 pypi
cfgv 3.5.0 pypi_0 pypi
charset-normalizer 3.4.4 pypi_0 pypi
click 8.2.1 pypi_0 pypi
cloudpickle 3.1.2 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
colorlog 6.10.1 pypi_0 pypi
comm 0.2.3 pyhe01879c_0 conda-forge
compressed-tensors 0.11.0 pypi_0 pypi
contourpy 1.3.2 pypi_0 pypi
cpm-kernels 1.0.11 pypi_0 pypi
crcmod 1.7 pypi_0 pypi
cryptography 46.0.3 pypi_0 pypi
cuda-bindings 13.0.3 pypi_0 pypi
cuda-pathfinder 1.3.2 pypi_0 pypi
cuda-python 13.0.3 pypi_0 pypi
cupy-cuda12x 13.6.0 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
dacite 1.9.2 pypi_0 pypi
datasets 3.3.2 pypi_0 pypi
debugpy 1.8.17 py310h25320af_0 conda-forge
decorator 5.2.1 pyhd8ed1ab_0 conda-forge
decord 0.6.0 pypi_0 pypi
decord2 2.0.0 pypi_0 pypi
deepspeed 0.16.9 pypi_0 pypi
depyf 0.19.0 pypi_0 pypi
dill 0.3.8 pypi_0 pypi
diskcache 5.6.3 pypi_0 pypi
distlib 0.4.0 pypi_0 pypi
distro 1.9.0 pypi_0 pypi
dnspython 2.8.0 pypi_0 pypi
docstring-parser 0.17.0 pypi_0 pypi
einops 0.8.1 pypi_0 pypi
email-validator 2.3.0 pypi_0 pypi
exceptiongroup 1.3.1 pyhd8ed1ab_0 conda-forge
executing 2.2.1 pyhd8ed1ab_0 conda-forge
expat 2.7.3 h3385a95_0
fastapi 0.122.0 pypi_0 pypi
fastapi-cli 0.0.16 pypi_0 pypi
fastapi-cloud-cli 0.5.1 pypi_0 pypi
fastar 0.7.0 pypi_0 pypi
fastrlock 0.8.3 pypi_0 pypi
ffmpy 1.0.0 pypi_0 pypi
filelock 3.20.0 pypi_0 pypi
flash-attn 2.8.3 pypi_0 pypi
flashinfer-cubin 0.5.2 pypi_0 pypi
flashinfer-python 0.5.2 pypi_0 pypi
fonttools 4.60.1 pypi_0 pypi
frozendict 2.4.7 pypi_0 pypi
frozenlist 1.8.0 pypi_0 pypi
fsspec 2024.12.0 pypi_0 pypi
future 1.0.0 pypi_0 pypi
gekko 1.3.0 pypi_0 pypi
gguf 0.17.1 pypi_0 pypi
gitdb 4.0.12 pypi_0 pypi
gitpython 3.1.45 pypi_0 pypi
gradio 5.32.1 pypi_0 pypi
gradio-client 1.10.2 pypi_0 pypi
graphviz 0.21 pypi_0 pypi
greenlet 3.3.0 pypi_0 pypi
groovy 0.1.2 pypi_0 pypi
grpcio 1.75.1 pypi_0 pypi
grpcio-health-checking 1.75.1 pypi_0 pypi
grpcio-reflection 1.75.1 pypi_0 pypi
grpcio-tools 1.75.1 pypi_0 pypi
h11 0.16.0 pypi_0 pypi
hf-transfer 0.1.9 pypi_0 pypi
hf-xet 1.2.0 pypi_0 pypi
hjson 3.1.0 pypi_0 pypi
httpcore 1.0.9 pypi_0 pypi
httptools 0.7.1 pypi_0 pypi
httpx 0.28.1 pypi_0 pypi
huggingface-hub 0.36.0 pypi_0 pypi
icecream 2.1.8 pypi_0 pypi
identify 2.6.15 pypi_0 pypi
idna 3.11 pypi_0 pypi
importlib-metadata 8.7.0 pyhe01879c_1 conda-forge
interegular 0.3.3 pypi_0 pypi
ipykernel 7.1.0 pyha191276_0 conda-forge
ipython 8.37.0 pyh8f84b5b_0 conda-forge
jedi 0.19.2 pyhd8ed1ab_1 conda-forge
jieba 0.42.1 pypi_0 pypi
jinja2 3.1.6 pypi_0 pypi
jiter 0.12.0 pypi_0 pypi
jmespath 0.10.0 pypi_0 pypi
joblib 1.5.2 pypi_0 pypi
json-repair 0.54.1 pypi_0 pypi
jsonlines 4.0.0 pypi_0 pypi
jsonschema 4.25.1 pypi_0 pypi
jsonschema-specifications 2025.9.1 pypi_0 pypi
jupyter_client 8.6.3 pyhd8ed1ab_1 conda-forge
jupyter_core 5.9.1 pyhc90fa1f_0 conda-forge
kaggle 1.7.4.5 pypi_0 pypi
kagglehub 0.3.13 pypi_0 pypi
keyutils 1.6.3 hb9d3cd8_0 conda-forge
kiwisolver 1.4.9 pypi_0 pypi
krb5 1.21.3 h659f571_0 conda-forge
lark 1.2.2 pypi_0 pypi
latex2sympy2-extended 1.10.2 pypi_0 pypi
lazy-loader 0.4 pypi_0 pypi
ld_impl_linux-64 2.44 h153f514_2
libedit 3.1.20250104 pl5321h7949ede_0 conda-forge
libffi 3.4.4 h6a678d5_1
libgcc 15.2.0 h69a1729_7
libgcc-ng 15.2.0 h166f726_7
libgomp 15.2.0 h4751f2c_7
libnsl 2.0.0 h5eee18b_0
librosa 0.11.0 pypi_0 pypi
libsodium 1.0.20 h4ab18f5_0 conda-forge
libstdcxx 15.2.0 h39759b7_7
libstdcxx-ng 15.2.0 hc03a8fd_7
libuuid 1.41.5 h5eee18b_0
libxcb 1.17.0 h9b100fa_0
libzlib 1.3.1 hb25bd0a_0
liger-kernel 0.6.4 pypi_0 pypi
lightgbm 4.6.0 pypi_0 pypi
llguidance 0.7.30 pypi_0 pypi
llvmlite 0.44.0 pypi_0 pypi
lm-format-enforcer 0.11.3 pypi_0 pypi
lmdeploy 0.8.0 pypi_0 pypi
loguru 0.7.3 pypi_0 pypi
lxml 6.0.2 pypi_0 pypi
mako 1.3.10 pypi_0 pypi
markdown 3.10 pypi_0 pypi
markdown-it-py 4.0.0 pypi_0 pypi
markupsafe 3.0.3 pypi_0 pypi
math-verify 0.8.0 pypi_0 pypi
matplotlib 3.10.7 pypi_0 pypi
matplotlib-inline 0.2.1 pyhd8ed1ab_0 conda-forge
mdurl 0.1.2 pypi_0 pypi
mistral-common 1.8.5 pypi_0 pypi
modelscope 1.32.0 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
ms-swift 3.12.0 pypi_0 pypi
msgpack 1.1.2 pypi_0 pypi
msgspec 0.20.0 pypi_0 pypi
multidict 6.7.0 pypi_0 pypi
multiprocess 0.70.16 pypi_0 pypi
narwhals 2.13.0 pypi_0 pypi
ncurses 6.5 h7934f7d_0
nest-asyncio 1.6.0 pyhd8ed1ab_1 conda-forge
networkx 3.4.2 pypi_0 pypi
ninja 1.13.0 pypi_0 pypi
nltk 3.9.2 pypi_0 pypi
nodeenv 1.9.1 pypi_0 pypi
numba 0.61.2 pypi_0 pypi
numpy 2.2.6 pypi_0 pypi
nvidia-cublas-cu12 12.8.4.1 pypi_0 pypi
nvidia-cuda-cupti-cu12 12.8.90 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.8.93 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.8.90 pypi_0 pypi
nvidia-cudnn-cu12 9.10.2.21 pypi_0 pypi
nvidia-cudnn-frontend 1.16.0 pypi_0 pypi
nvidia-cufft-cu12 11.3.3.83 pypi_0 pypi
nvidia-cufile-cu12 1.13.1.3 pypi_0 pypi
nvidia-curand-cu12 10.3.9.90 pypi_0 pypi
nvidia-cusolver-cu12 11.7.3.90 pypi_0 pypi
nvidia-cusparse-cu12 12.5.8.93 pypi_0 pypi
nvidia-cusparselt-cu12 0.7.1 pypi_0 pypi
nvidia-cutlass-dsl 4.2.1 pypi_0 pypi
nvidia-ml-py 13.580.82 pypi_0 pypi
nvidia-nccl-cu12 2.27.3 pypi_0 pypi
nvidia-nvjitlink-cu12 12.8.93 pypi_0 pypi
nvidia-nvshmem-cu12 3.3.20 pypi_0 pypi
nvidia-nvtx-cu12 12.8.90 pypi_0 pypi
nvitop 1.6.0 pypi_0 pypi
omegaconf 2.3.0 pypi_0 pypi
openai 2.8.1 pypi_0 pypi
openai-harmony 0.0.4 pypi_0 pypi
opencv-python-headless 4.12.0.88 pypi_0 pypi
openssl 3.6.0 h26f9b46_0 conda-forge
optimum 2.0.0 pypi_0 pypi
optuna 4.6.0 pypi_0 pypi
orjson 3.11.4 pypi_0 pypi
oss2 2.19.1 pypi_0 pypi
outlines 0.1.11 pypi_0 pypi
outlines-core 0.2.11 pypi_0 pypi
packaging 25.0 pyh29332c3_1 conda-forge
pandas 2.3.3 pypi_0 pypi
parso 0.8.5 pyhcf101f3_0 conda-forge
partial-json-parser 0.2.1.1.post7 pypi_0 pypi
peft 0.15.2 pypi_0 pypi
pexpect 4.9.0 pyhd8ed1ab_1 conda-forge
pickleshare 0.7.5 pyhd8ed1ab_1004 conda-forge
pillow 11.3.0 pypi_0 pypi
pip 25.3 pyhc872135_0
platformdirs 4.5.0 pyhcf101f3_0 conda-forge
plotly 6.5.0 pypi_0 pypi
pooch 1.8.2 pypi_0 pypi
pre-commit 4.5.0 pypi_0 pypi
prometheus-client 0.23.1 pypi_0 pypi
prometheus-fastapi-instrumentator 7.1.0 pypi_0 pypi
prompt-toolkit 3.0.52 pyha770c72_0 conda-forge
propcache 0.4.1 pypi_0 pypi
protobuf 6.33.1 pypi_0 pypi
psutil 7.1.3 py310h139afa4_0 conda-forge
pthread-stubs 0.3 h0ce48e5_1
ptyprocess 0.7.0 pyhd8ed1ab_1 conda-forge
pure_eval 0.2.3 pyhd8ed1ab_1 conda-forge
py-cpuinfo 9.0.0 pypi_0 pypi
py-spy 0.4.1 pypi_0 pypi
pyarrow 22.0.0 pypi_0 pypi
pybase64 1.4.2 pypi_0 pypi
pycodestyle 2.14.0 pypi_0 pypi
pycountry 24.6.1 pypi_0 pypi
pycparser 2.23 pypi_0 pypi
pycryptodome 3.23.0 pypi_0 pypi
pycryptodomex 3.23.0 pypi_0 pypi
pydantic 2.11.10 pypi_0 pypi
pydantic-core 2.33.2 pypi_0 pypi
pydantic-extra-types 2.10.6 pypi_0 pypi
pydub 0.25.1 pypi_0 pypi
pygments 2.19.2 pyhd8ed1ab_0 conda-forge
pyparsing 3.2.5 pypi_0 pypi
pyproject-hooks 1.2.0 pypi_0 pypi
python 3.10.19 h6fa692b_0
python-dateutil 2.9.0.post0 pyhe01879c_2 conda-forge
python-dotenv 1.2.1 pypi_0 pypi
python-json-logger 4.0.0 pypi_0 pypi
python-multipart 0.0.20 pypi_0 pypi
python-slugify 8.0.4 pypi_0 pypi
python_abi 3.10 2_cp310 conda-forge
pytz 2025.2 pypi_0 pypi
pyyaml 6.0.3 pypi_0 pypi
pyzmq 27.1.0 py310h4f33d48_0 conda-forge
qwen-omni-utils 0.0.8 pypi_0 pypi
qwen-vl-utils 0.0.14 pypi_0 pypi
ray 2.52.0 pypi_0 pypi
readline 8.3 hc2a1206_0
referencing 0.37.0 pypi_0 pypi
regex 2025.11.3 pypi_0 pypi
requests 2.32.5 pypi_0 pypi
rich 14.2.0 pypi_0 pypi
rich-toolkit 0.16.0 pypi_0 pypi
rignore 0.7.6 pypi_0 pypi
rouge 1.0.1 pypi_0 pypi
rpds-py 0.29.0 pypi_0 pypi
ruff 0.14.6 pypi_0 pypi
safehttpx 0.1.7 pypi_0 pypi
safetensors 0.7.0 pypi_0 pypi
scikit-learn 1.7.2 pypi_0 pypi
scipy 1.15.3 pypi_0 pypi
semantic-version 2.10.0 pypi_0 pypi
sentencepiece 0.2.1 pypi_0 pypi
sentry-sdk 2.46.0 pypi_0 pypi
setproctitle 1.3.7 pypi_0 pypi
setuptools 80.9.0 py310h06a4308_0
sgl-kernel 0.3.17.post1 pypi_0 pypi
sglang 0.5.5.post3 pypi_0 pypi
shellingham 1.5.4 pypi_0 pypi
simplejson 3.20.2 pypi_0 pypi
six 1.17.0 pyhe01879c_1 conda-forge
smmap 5.0.2 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
socksio 1.0.0 pypi_0 pypi
sortedcontainers 2.4.0 pypi_0 pypi
soundfile 0.13.1 pypi_0 pypi
soxr 1.0.0 pypi_0 pypi
sqlalchemy 2.0.44 pypi_0 pypi
sqlite 3.51.0 h2a70700_0
stack_data 0.6.3 pyhd8ed1ab_1 conda-forge
starlette 0.50.0 pypi_0 pypi
sympy 1.14.0 pypi_0 pypi
tabulate 0.9.0 pypi_0 pypi
tensorboard 2.20.0 pypi_0 pypi
tensorboard-data-server 0.7.2 pypi_0 pypi
text-unidecode 1.3 pypi_0 pypi
threadpoolctl 3.6.0 pypi_0 pypi
tiktoken 0.12.0 pypi_0 pypi
timeout-decorator 0.5.0 pypi_0 pypi
timm 1.0.22 pypi_0 pypi
tk 8.6.15 h54e0aa7_0
tokenizers 0.22.1 pypi_0 pypi
tomli 2.3.0 pypi_0 pypi
tomlkit 0.13.3 pypi_0 pypi
torch 2.8.0 pypi_0 pypi
torch-memory-saver 0.0.9 pypi_0 pypi
torchao 0.9.0 pypi_0 pypi
torchaudio 2.8.0 pypi_0 pypi
torchvision 0.23.0 pypi_0 pypi
tornado 6.5.2 py310h7c4b9e2_2 conda-forge
tqdm 4.67.1 pypi_0 pypi
traitlets 5.14.3 pyhd8ed1ab_1 conda-forge
transformers 4.57.1 pypi_0 pypi
transformers-stream-generator 0.0.5 pypi_0 pypi
triton 3.4.0 pypi_0 pypi
trl 0.20.0 pypi_0 pypi
typer 0.20.0 pypi_0 pypi
typing-inspection 0.4.2 pypi_0 pypi
typing_extensions 4.15.0 pyhcf101f3_0 conda-forge
tzdata 2025.2 pypi_0 pypi
urllib3 2.5.0 pypi_0 pypi
uvicorn 0.38.0 pypi_0 pypi
uvloop 0.21.0 pypi_0 pypi
virtualenv 20.35.4 pypi_0 pypi
vllm 0.11.0 pypi_0 pypi
wandb 0.23.0 pypi_0 pypi
watchfiles 1.1.1 pypi_0 pypi
wcwidth 0.2.14 pyhd8ed1ab_0 conda-forge
webencodings 0.5.1 pypi_0 pypi
websockets 15.0.1 pypi_0 pypi
werkzeug 3.1.3 pypi_0 pypi
wheel 0.45.1 py310h06a4308_0
xformers 0.0.32.post1 pypi_0 pypi
xgboost 3.1.2 pypi_0 pypi
xgrammar 0.1.25 pypi_0 pypi
xorg-libx11 1.8.12 h9b100fa_1
xorg-libxau 1.0.12 h9b100fa_0
xorg-libxdmcp 1.1.5 h9b100fa_0
xorg-xorgproto 2024.1 h5eee18b_1
xxhash 3.6.0 pypi_0 pypi
xz 5.6.4 h5eee18b_1
yarl 1.22.0 pypi_0 pypi
zeromq 4.3.5 h387f397_9 conda-forge
zipp 3.23.0 pyhd8ed1ab_0 conda-forge
zlib 1.3.1 hb25bd0a_0
zstandard 0.25.0 pypi_0 pypi

Additional context
Add any other context about the problem here(在这里补充其他信息)

Start with the custom twt_scheduler step method shown in the issue and the scripts/rlhf_ds.py GRPO launch configuration. Reproduce the empty rollout case with the provided swift rollout and deepspeed commands, then trace the correction path; done means the multi-turn rollout no longer errors after an empty output is handled.