grpo训练400step整体性能变差，KL骤增

grpo训练过程前期reward上升趋势平稳，为什么到400step的时候3个reward同时下降？KL骤增，lr到400步不再变化；以下是我的训练脚本，请问是什么问题，应该如何修改？感谢！ `export MASTER_PORT=29501 MAX_PIXELS=602112 \ NPROC_PER_NODE=4 \ swift rlhf \ --rlhf_type grpo \ --model /home/qid/lgs/lgs-foodkg/ms-swift/output_kg/v2-20251106-104053/checkpoint-2125-merged \ --train_type lora \ --dataset '/home/qid/lgs/lgs-foodkg/ms-swift/data/grpo_train_entities_full.json' \ --use_vllm true \ --external_plugins examples/train/grpo/plugin/plugin_kg.py \ --vllm_mode colocate \ --vllm_gpu_memory_utilization 0.5 \ --vllm_tensor_parallel_size 4 \ --torch_dtype bfloat16 \ --system /home/qid/lgs/lgs-foodkg/ms-swift/examples/train/grpo/plugin/kg_prompt.txt \ --num_train_epochs 1 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --split_dataset_ratio 0 \ --learning_rate 5e-6 \ --save_total_limit 2 \ --logging_steps 1 \ --output_dir output_grpo \ --gradient_accumulation_steps 4 \ --warmup_ratio 0.05 \ --dataloader_num_workers 4 \ --max_completion_length 4096 \ --reward_funcs entity relation format \ --reward_weights 1 1 0.1 \ --num_generations 8 \ --sleep_level 1 \ --save_steps 100 \ --temperature 1.0 \ --top_p 0.85 \ --gradient_checkpointing_kwargs '{"use_reentrant": true}' \ --max_length 8192 \ --lora_rank 128 \ --lora_alpha 256 \ --attn_impl flash_attn \ --deepspeed zero3 \ --top_k 50 \ --move_model_batches 16 \ --report_to wandb `

Start by running the supplied swift rlhf command with examples/train/grpo/plugin/plugin_kg.py and inspect the logged reward, KL, and learning-rate values around step 400. Compare those metrics with the command's training settings, then rerun the same training after a proposed configuration change and verify whether the degradation is resolved.