2d5-7b : I found the LoRA-checkpoint saved with multiple gpu is incorrect

I found with 2d5-7b the checkpoint saved from LoRA tuning finetune.py with one GPU is correct, while with multiple GPU the model saved is incorrect.

Does anyone met similar problem?

For example I saved LoRA-checkpoint `multi` with 2 GPU training and saved `single` with 1 GPU training
**With multiple GPUs**
```
==== Model merged successfully from checkpoint: ./multi
==== Model merged successfully from checkpoint: ./multi

trainable params: 151,003,136 || all params: 11,246,729,216 || trainable%: 1.3426
init mix data at rank 1
load 20 data
load 10 data
load 10 data
10samples is loaded
True
trainable params: 151,003,136 || all params: 11,246,729,216 || trainable%: 1.3426
Loading data...
Load 20 samples from ['data/only_text_example.json', '0.02']
Load 10 samples from ['data/single_turn_single_image_example.json', '0.01']
Load 10 samples from ['data/multi_turn_multi_images_example.json', '0.01']
init mix data at rank 0
load 20 data
load 10 data
load 10 data
10samples is loaded
True
[2024-08-19 04:49:00,958] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter stage3_gather_fp16_weights_on_model_save is deprecated use gather_16bit_weights_on_model_save instead
[2024-08-19 04:49:00,960] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter stage3_gather_fp16_weights_on_model_save is deprecated use gather_16bit_weights_on_model_save instead
0%| | 0/10 [00:00