单机多卡训练-显存负载不均衡

4张卡 训练中途报cuda out of memory， 4张卡会存在2张卡显存打满，其他两张空余很多， 目前通过减小batchsize 方式去控制，请问有什么方法能平衡各张卡的负载？

nproc_per_node=4
NPROC_PER_NODE=$nproc_per_node
swift sft
--model models/jina-code-embeddings-1.5b
--task_type embedding
--model_type qwen2_5
--train_type full
--dataset emd_train_data/train_data_v1.0/data_v1.0/train_and_val_data.json
--split_dataset_ratio 0.05
--eval_strategy steps
--output_dir output/train_v1.0/jina-1.5b
--eval_steps 2000
--num_train_epochs 3
--save_steps 2000
--per_device_train_batch_size 8
--per_device_eval_batch_size 4
--gradient_accumulation_steps 4
--learning_rate 1e-5
--loss_type infonce
--label_names labels
--dataloader_drop_last true
--deepspeed zero2

Start with the provided swift sft command and its four-process configuration, then inspect how this training setup distributes batches and model memory across GPUs. Reproduce the reported run while monitoring all four cards; done means the cause of the imbalance is identified and the training no longer hits CUDA out-of-memory on only two cards.