GRPO训练，自定义奖励函数missing required positional arguments: 'clean_response' and 'key_response'

报错信息：
[rank3]: Traceback (most recent call last):
[rank3]: File "/root/projects/omni/ms-swift-main/swift/cli/rlhf.py", line 7, in
[rank3]: rlhf_main()
[rank3]: File "/root/projects/omni/ms-swift-main/swift/llm/train/rlhf.py", line 233, in rlhf_main
[rank3]: return SwiftRLHF(args).main()
[rank3]: File "/root/projects/omni/ms-swift-main/swift/llm/base.py", line 49, in main
[rank3]: result = self.run()
[rank3]: File "/root/projects/omni/ms-swift-main/swift/ray/base.py", line 170, in wrapper
[rank3]: return func(self, *args, **kwargs)
[rank3]: File "/root/projects/omni/ms-swift-main/swift/llm/train/sft.py", line 196, in run
[rank3]: return self.train(trainer)
[rank3]: File "/root/projects/omni/ms-swift-main/swift/llm/train/sft.py", line 244, in train
[rank3]: trainer.train(trainer.args.resume_from_checkpoint)
[rank3]: File "/root/projects/omni/ms-swift-main/swift/trainers/mixin.py", line 816, in train
[rank3]: res = super().train(*args, **kwargs)
[rank3]: File "/root/miniconda3/envs/omni/lib/python3.10/site-packages/transformers/trainer.py", line 2328, in train
[rank3]: return inner_training_loop(
[rank3]: File "/root/miniconda3/envs/omni/lib/python3.10/site-packages/transformers/trainer.py", line 2672, in _inner_training_loop
[rank3]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank3]: File "/root/projects/omni/ms-swift-main/swift/trainers/rlhf_trainer/grpo_trainer.py", line 1553, in training_step
[rank3]: return super().training_step(model, inputs, num_items_in_batch)
[rank3]: File "/root/miniconda3/envs/omni/lib/python3.10/site-packages/transformers/trainer.py", line 4003, in training_step
[rank3]: inputs = self._prepare_inputs(inputs)
[rank3]: File "/root/projects/omni/ms-swift-main/swift/trainers/rlhf_trainer/utils.py", line 407, in wrapper
[rank3]: return func(self, *args, **kwargs)
[rank3]: File "/root/projects/omni/ms-swift-main/swift/trainers/rlhf_trainer/grpo_trainer.py", line 177, in _prepare_inputs
[rank3]: generation_batch = self._generate_and_score_completions(generation_batch)
[rank3]: File "/root/projects/omni/ms-swift-main/swift/trainers/rlhf_trainer/utils.py", line 407, in wrapper
[rank3]: return func(self, *args, **kwargs)
[rank3]: File "/root/projects/omni/ms-swift-main/swift/trainers/rlhf_trainer/grpo_trainer.py", line 221, in _generate_and_score_completions
[rank3]: total_rewards_per_func = self._score_completions(inputs)
[rank3]: File "/root/projects/omni/ms-swift-main/swift/trainers/rlhf_trainer/utils.py", line 407, in wrapper
[rank3]: return func(self, *args, **kwargs)
[rank3]: File "/root/projects/omni/ms-swift-main/swift/trainers/rlhf_trainer/grpo_trainer.py", line 309, in _score_completions
[rank3]: local_rewards_per_func = self._compute_rewards_per_func(inputs)
[rank3]: File "/root/projects/omni/ms-swift-main/swift/trainers/rlhf_trainer/grpo_trainer.py", line 342, in _compute_rewards_per_func
[rank3]: output_reward_func = reward_func(completions, **reward_kwargs)
[rank3]: TypeError: asrORM.call() missing 2 required positional arguments: 'clean_response' and 'key_response'

swift版本：
Name: ms_swift
Version: 3.11.0.dev0
Summary: Swift: Scalable lightWeight Infrastructure for Fine-Tuning
Home-page: https://github.com/modelscope/swift
Author: DAMO ModelScope teams
Author-email: contact@modelscope.cn
License: Apache License 2.0
Location: /root/projects/omni/ms-swift-main
Editable project location: /root/projects/omni/ms-swift-main
Requires: accelerate, addict, aiohttp, attrdict, binpacking, charset_normalizer, cpm_kernels, dacite, datasets, einops, fastapi, gradio, importlib_metadata, jieba, json_repair, matplotlib, modelscope, nltk, numpy, omegaconf, openai, oss2, pandas, peft, pillow, PyYAML, requests, rouge, safetensors, scipy, sentencepiece, simplejson, sortedcontainers, tensorboard, tiktoken, tqdm, transformers, transformers_stream_generator, trl, uvicorn, zstandard

训练脚本：
MAX_PIXELS=0
NPROC_PER_NODE=7
ENABLE_AUDIO_OUTPUT=False
CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7
swift rlhf
--model {隐藏}
--dataset {隐藏}
--val_dataset {隐藏}
--output_dir {隐藏}
--seed 42
--rlhf_type grpo
--advantage_estimator grpo
--scale_rewards batch
--importance_sampling_level sequence
--external_plugins /root/projects/omni/plugin.py
--reward_funcs external_format external_asr external_judge
--reward_weights 1.0 1.0 1.0
--train_type full
--torch_dtype bfloat16
--max_length 2048
--max_completion_length 256
--num_train_epochs 10
--per_device_train_batch_size 4
--per_device_eval_batch_size 4
--learning_rate 1e-5
--gradient_accumulation_steps 8
--save_strategy epoch
--eval_strategy epoch
--eval_steps 200
--save_steps 200
--save_total_limit 20
--logging_steps 20
--warmup_ratio 0.05
--dataloader_num_workers 4
--dataset_num_proc 4
--num_generations 4
--temperature 1.0
--top_p 1.0
--top_k 32768
--log_completions True
--save_only_model True
--use_vllm True
--vllm_mode server
--vllm_server_base_url http://0.0.0.0:8000
--vllm_server_timeout 300
--async_generate True \

在用GRPO微调qwen2.5-omni去做ASR任务，第一个epoch跑到60%左右会报这个错误。尝试debug了一下但找不到问题在哪。因为和这个自定义奖励函数内部没什么关系，这里就没贴上来。

1. 能跑完60%，但目前跑了两次发现貌似就会在60%这个点抱错，用出错的数据集的子集跑可以顺利跑完，但一用全量的就不行了。原本怀疑是数据有问题，但也过了一遍数据，可以确定每个数据都有"clean_response"和"key_response"这两个字段，但貌似在某个step这两个字段被移除了？removed_unused_columns设置的false
2. 之前用更小的数据集，async_generate = False, scale_rewards = group, importance_sampling_level = token能顺利跑完。现在修改了这几个参数，但不确定和这个bug有没有关系

求大佬们帮帮

Start at swift/trainers/rlhf_trainer/grpo_trainer.py, especially _compute_rewards_per_func and the _score_completions path shown in the traceback. Reproduce with the full dataset and the listed async_generate, reward-scaling, and importance-sampling settings, then inspect which fields reach the custom reward function at the failing step. Done means the full training run completes while clean_response and key_response remain available to the reward function.